From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 25B54A04FF;
Tue, 24 May 2022 05:36:07 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 0E76742B6E;
Tue, 24 May 2022 05:35:42 +0200 (CEST)
Received: from mga14.intel.com (mga14.intel.com [192.55.52.115])
by mails.dpdk.org (Postfix) with ESMTP id 480DF42B75
for <dev@dpdk.org>; Tue, 24 May 2022 05:35:40 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;
d=intel.com; i=@intel.com; q=dns/txt; s=Intel;
t=1653363340; x=1684899340;
h=from:to:cc:subject:date:message-id:in-reply-to: references;
bh=jVIUbG9caZJ27p7oaD98uVLiyYuLObHsUW5q7s9tQSs=;
b=mbmcCniE/hB2tWOwfUVVnhdLQJzf0oYA9KUfjarGs0BdH7hZmeqKNUGD
+bGKjsyq+jta4Gxd0SHq/WKoqujCCBTFVuz6a2vjVTBj+Qfc52FhXK2DW
cgcDsDryRcWetwv3mV58QQ2hSP52Ln1nybnlv0hkxbAQwp9j/CBYiL/Bh
UwCiJrTMSBoCy8n2CM5bMkV+Rdk6nIj8RXT+ZW6SGi8jXD5kArVHYjQQ7
9s2l2MpIdKxd6he449t1HfqBpZhZpm8tp8V/DVBHssiMJZyAlsfdeKdUG
Vvr+3AbBhjq7sIBRC18CcBUGsbZ0ph6UHhN9z472u+bKQJGcmJzDYltu6 A==;
X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533621"
X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533621"
Received: from orsmga005.jf.intel.com ([10.7.209.41])
by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
23 May 2022 20:35:39 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745046971"
Received: from dpdk-dipei.sh.intel.com ([10.67.110.238])
by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:37 -0700
From: Andy Pei <andy.pei@intel.com>
To: dev@dpdk.org
Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com,
changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com
Subject: [PATCH v10 06/13] vdpa/ifc: add block device SW live-migration
Date: Tue, 24 May 2022 10:48:10 +0800
Message-Id: <1653360497-18080-7-git-send-email-andy.pei@intel.com>
X-Mailer: git-send-email 1.8.3.1
In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com>
References: <1643093258-47258-2-git-send-email-andy.pei@intel.com>
<1653360497-18080-1-git-send-email-andy.pei@intel.com>
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Add SW live-migration support to block device.
For block device, it is critical that no packet
should be dropped. So when virtio blk device is
paused, make sure hardware last_avail_idx and
last_used_idx are the same. This indicates all
requests have received acks, and no inflight IO.
Signed-off-by: Andy Pei <andy.pei@intel.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
---
drivers/vdpa/ifc/base/ifcvf.h | 1 +
drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++----
2 files changed, 39 insertions(+), 4 deletions(-)
diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h
index b6fdfdb..9d95aac 100644
--- a/drivers/vdpa/ifc/base/ifcvf.h
+++ b/drivers/vdpa/ifc/base/ifcvf.h
@@ -63,6 +63,7 @@
#define IFCVF_MEDIATED_VRING 0x200000000000
#define IFCVF_32_BIT_MASK 0xffffffff
+#define IFCVF_16_BIT_MASK 0xffff
#ifndef VHOST_USER_PROTOCOL_F_CONFIG
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index 61fb427..0f9db8a 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -316,8 +316,34 @@ struct rte_vdpa_dev_info {
uint64_t features = 0;
uint64_t log_base = 0, log_size = 0;
uint64_t len;
+ u32 ring_state = 0;
vid = internal->vid;
+
+ /* to make sure no packet is lost for blk device
+ * do not stop until last_avail_idx == last_used_idx
+ */
+ if (internal->hw.device_type == IFCVF_BLK) {
+ for (i = 0; i < hw->nr_vring; i++) {
+ do {
+ if (hw->lm_cfg != NULL)
+ ring_state = *(u32 *)(hw->lm_cfg +
+ IFCVF_LM_RING_STATE_OFFSET +
+ i * IFCVF_LM_CFG_SIZE);
+ hw->vring[i].last_avail_idx =
+ (u16)(ring_state & IFCVF_16_BIT_MASK);
+ hw->vring[i].last_used_idx =
+ (u16)(ring_state >> 16);
+ if (hw->vring[i].last_avail_idx !=
+ hw->vring[i].last_used_idx) {
+ ifcvf_notify_queue(hw, i);
+ usleep(10);
+ }
+ } while (hw->vring[i].last_avail_idx !=
+ hw->vring[i].last_used_idx);
+ }
+ }
+
ifcvf_stop_hw(hw);
for (i = 0; i < hw->nr_vring; i++)
@@ -641,8 +667,10 @@ struct rte_vdpa_dev_info {
}
hw->vring[i].avail = gpa;
- /* Direct I/O for Tx queue, relay for Rx queue */
- if (i & 1) {
+ /* NET: Direct I/O for Tx queue, relay for Rx queue
+ * BLK: relay every queue
+ */
+ if ((internal->hw.device_type == IFCVF_NET) && (i & 1)) {
gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used);
if (gpa == 0) {
DRV_LOG(ERR, "Fail to get GPA for used ring.");
@@ -692,8 +720,12 @@ struct rte_vdpa_dev_info {
for (i = 0; i < hw->nr_vring; i++) {
/* synchronize remaining new used entries if any */
- if ((i & 1) == 0)
+ if (internal->hw.device_type == IFCVF_NET) {
+ if ((i & 1) == 0)
+ update_used_ring(internal, i);
+ } else if (internal->hw.device_type == IFCVF_BLK) {
update_used_ring(internal, i);
+ }
rte_vhost_get_vhost_vring(vid, i, &vq);
len = IFCVF_USED_RING_LEN(vq.size);
@@ -755,7 +787,9 @@ struct rte_vdpa_dev_info {
}
}
- for (qid = 0; qid < q_num; qid += 2) {
+ for (qid = 0; qid < q_num; qid += 1) {
+ if ((internal->hw.device_type == IFCVF_NET) && (qid & 1))
+ continue;
ev.events = EPOLLIN | EPOLLPRI;
/* leave a flag to mark it's for interrupt */
ev.data.u64 = 1 | qid << 1 |
--
1.8.3.1