From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 17134A054F; Tue, 16 Mar 2021 13:25:17 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D06094069F; Tue, 16 Mar 2021 13:25:16 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mails.dpdk.org (Postfix) with ESMTP id 4793F4003D for ; Tue, 16 Mar 2021 13:25:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1615897514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=od3Nnt2sYZ6/ekDeG4Pr027UCjGELzJcPel0PCsAUvY=; b=GrgjW1OnWJGsvys2TOob3m/A2yunN1cfwPiJsdSUBGYwuy5ze11KKtaJn+rgS53/5d8F27 h9DkRvxCxZjUICm7RZiYOW8N0iraSksnDpTSb+/okcuxBMmscxk1Gp5Z0eX6Ww39QYJzdi mOGkEYln3eqBm731J/WW6obL+z3XWyQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-426-QtJiUV1ZMSuRVWkSYuy6NQ-1; Tue, 16 Mar 2021 08:25:12 -0400 X-MC-Unique: QtJiUV1ZMSuRVWkSYuy6NQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AE9E819057A8; Tue, 16 Mar 2021 12:25:11 +0000 (UTC) Received: from [10.36.110.35] (unknown [10.36.110.35]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BCFA71037E8C; Tue, 16 Mar 2021 12:25:10 +0000 (UTC) To: "Xia, Chenbo" , "dev@dpdk.org" , "amorenoz@redhat.com" References: <20201222135658.53916-1-maxime.coquelin@redhat.com> <20201222135658.53916-3-maxime.coquelin@redhat.com> From: Maxime Coquelin Message-ID: <01587854-a915-c17b-ae48-6bfa45590af3@redhat.com> Date: Tue, 16 Mar 2021 13:25:08 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH 2/3] vhost: move dirty logging cache out of the virtqueue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Chenbo, On 1/8/21 8:20 AM, Xia, Chenbo wrote: > Hi Maxime, > >> -----Original Message----- >> From: Maxime Coquelin >> Sent: Tuesday, December 22, 2020 9:57 PM >> To: dev@dpdk.org; Xia, Chenbo ; amorenoz@redhat.com >> Cc: Maxime Coquelin >> Subject: [PATCH 2/3] vhost: move dirty logging cache out of the virtqueue >> >> This patch moves the per-virtqueue's dirty logging cache >> out of the virtqueue struct, by allocating it dynamically >> only when live-migration is enabled. >> >> It saves 8 cachelines in vhost_virtqueue struct. >> >> Signed-off-by: Maxime Coquelin >> --- >> lib/librte_vhost/vhost.c | 12 ++++++++++++ >> lib/librte_vhost/vhost.h | 2 +- >> lib/librte_vhost/vhost_user.c | 25 +++++++++++++++++++++++++ >> 3 files changed, 38 insertions(+), 1 deletion(-) >> >> diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c >> index 4e5df862aa..ec6459b2d1 100644 >> --- a/lib/librte_vhost/vhost.c >> +++ b/lib/librte_vhost/vhost.c >> @@ -144,6 +144,10 @@ __vhost_log_cache_sync(struct virtio_net *dev, struct >> vhost_virtqueue *vq) >> if (unlikely(!dev->log_base)) >> return; >> >> + /* No cache, nothing to sync */ >> + if (unlikely(!vq->log_cache)) >> + return; >> + >> rte_smp_wmb(); >> >> log_base = (unsigned long *)(uintptr_t)dev->log_base; >> @@ -176,6 +180,14 @@ vhost_log_cache_page(struct virtio_net *dev, struct >> vhost_virtqueue *vq, >> uint32_t offset = page / (sizeof(unsigned long) << 3); >> int i; >> >> + if (unlikely(!vq->log_cache)) { >> + /* No logging cache allocated, write dirty log map directly */ >> + rte_smp_wmb(); >> + vhost_log_page((uint8_t *)(uintptr_t)dev->log_base, page); >> + >> + return; >> + } >> + >> for (i = 0; i < vq->log_cache_nb_elem; i++) { >> struct log_cache_entry *elem = vq->log_cache + i; >> >> diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h >> index d132e4ae54..e2f14034b4 100644 >> --- a/lib/librte_vhost/vhost.h >> +++ b/lib/librte_vhost/vhost.h >> @@ -183,7 +183,7 @@ struct vhost_virtqueue { >> bool used_wrap_counter; >> bool avail_wrap_counter; >> >> - struct log_cache_entry log_cache[VHOST_LOG_CACHE_NR]; >> + struct log_cache_entry *log_cache; >> uint16_t log_cache_nb_elem; >> >> rte_rwlock_t iotlb_lock; >> diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c >> index 45c8ac09da..7ac3963a07 100644 >> --- a/lib/librte_vhost/vhost_user.c >> +++ b/lib/librte_vhost/vhost_user.c >> @@ -1978,6 +1978,11 @@ vhost_user_get_vring_base(struct virtio_net **pdev, >> rte_free(vq->batch_copy_elems); >> vq->batch_copy_elems = NULL; >> >> + if (vq->log_cache) { >> + rte_free(vq->log_cache); >> + vq->log_cache = NULL; >> + } >> + > > I think we'd better check and free the log cache in function free_vq() too? > It is possible that during migration, vhost destroys, right? Correct, I will do it too in free_vq(). Thanks! Maxime > Thanks, > Chenbo > >> msg->size = sizeof(msg->payload.state); >> msg->fd_num = 0; >> >> @@ -2077,6 +2082,7 @@ vhost_user_set_log_base(struct virtio_net **pdev, struct >> VhostUserMsg *msg, >> int fd = msg->fds[0]; >> uint64_t size, off; >> void *addr; >> + uint32_t i; >> >> if (validate_msg_fds(msg, 1) != 0) >> return RTE_VHOST_MSG_RESULT_ERR; >> @@ -2130,6 +2136,25 @@ vhost_user_set_log_base(struct virtio_net **pdev, >> struct VhostUserMsg *msg, >> dev->log_base = dev->log_addr + off; >> dev->log_size = size; >> >> + for (i = 0; i < dev->nr_vring; i++) { >> + struct vhost_virtqueue *vq = dev->virtqueue[i]; >> + >> + if (vq->log_cache) { >> + rte_free(vq->log_cache); >> + vq->log_cache = NULL; >> + } >> + vq->log_cache_nb_elem = 0; >> + vq->log_cache = rte_zmalloc("vq log cache", >> + sizeof(struct log_cache_entry) * VHOST_LOG_CACHE_NR, >> + 0); >> + /* >> + * If log cache alloc fail, don't fail migration, but no >> + * caching will be done, which will impact performance >> + */ >> + if (!vq->log_cache) >> + VHOST_LOG_CONFIG(ERR, "Failed to allocate VQ logging >> cache\n"); >> + } >> + >> /* >> * The spec is not clear about it (yet), but QEMU doesn't expect >> * any payload in the reply. >> -- >> 2.29.2 >