From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 70118A04FA; Wed, 5 Feb 2020 22:04:24 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 73F631C1E1; Wed, 5 Feb 2020 22:04:23 +0100 (CET) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by dpdk.org (Postfix) with ESMTP id 2BF901C1CA for ; Wed, 5 Feb 2020 22:04:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580936661; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5OYwWYN2lGfMgFo42TG0sGStyzngvek141/832n3uPM=; b=Bj39sCa9jz1aa5oMFai7RhdMqM7HL4p860HFmxQu/MvnIlgsqb2OjaZA3/6+mgPVzGo65m fAIjestZkNBwKdq+JmrQajp7raUMJm1vsdFptuXSdzynzTkFZcIo1X5a1WXm0ptL4g1QMd hcE9SGDwxJUm0uDFgW4UknE1G4HT6oE= Received: from mail-vk1-f198.google.com (mail-vk1-f198.google.com [209.85.221.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-414-1FQJZnhfNOeBPjTN231hig-1; Wed, 05 Feb 2020 16:04:19 -0500 Received: by mail-vk1-f198.google.com with SMTP id w23so1107975vke.15 for ; Wed, 05 Feb 2020 13:04:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gk4wXk2HLrBSqOsO/fg9tXwVP9lSILj3f6YWrPlK9Lw=; b=ng17z+XusjthsJ4Rsl/y3y2tJ1IcPuBIinfY7nVTf3SZrlXoTk/IWSPzCNPNAGjptf 5gPl46Q0pKHOf7U25vsth5dwQKGbJ4uqfs8AxwpcLUdB9NT7wQbrTla8IQQZbKI8gD9G P5fCLB3YX6OOEn7zBeHkAPXL1b9kEwfM/sjoeI5tT7qphoILD6utZ0A7d7wslhmvpxFQ I22gGmEAp+n9jymrmy+FLLann/Z7JEaMy387t0M7MFpDgYNbj1HtDpwhqoiz4sObRLim kxf6wm5imlY3CSWtzIcw7lLh031ZmmQfAKz/PvAXUDjHt+8bxLLquDswJuXZXRuk2P5G t2cQ== X-Gm-Message-State: APjAAAUsswbFlykgI5xVEythlNewemqrnaonKXXGJsHywzFkD0AgOfak N2ATM1sugrL++vxlaMI+QFdW7L6zbXKhOjSR8E5eZgax7iO9/ngARw4q0u+O/lvnpvA/Zw2l7dt vzlbI1ePQfGXonVymIlQ= X-Received: by 2002:a1f:72c3:: with SMTP id n186mr2883055vkc.12.1580936658759; Wed, 05 Feb 2020 13:04:18 -0800 (PST) X-Google-Smtp-Source: APXvYqw5DjgjfPy+/2fnSdCeWivcQ8a7+vwPCa8TW6iOfdpcPMmq8IxChiumTuenuqcW75bkfs0qkJjcwcZYS9JY2xs= X-Received: by 2002:a1f:72c3:: with SMTP id n186mr2883028vkc.12.1580936658329; Wed, 05 Feb 2020 13:04:18 -0800 (PST) MIME-Version: 1.0 References: <20200117042555.22567-1-tyos@jp.ibm.com> In-Reply-To: <20200117042555.22567-1-tyos@jp.ibm.com> From: David Marchand Date: Wed, 5 Feb 2020 22:04:07 +0100 Message-ID: To: Takeshi Yoshimura Cc: dev , drc@ibm.com, dpdk stable , David Christensen X-MC-Unique: 1FQJZnhfNOeBPjTN231hig-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [PATCH] vfio: fix VFIO mapping failures in ppc64le X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Fri, Jan 17, 2020 at 5:26 AM Takeshi Yoshimura wrote: > > ppc64le failed when using large physical memory. I found problems in my t= wo > commits in the past. Let's hope this is not a "Jamais deux sans trois" ;-). Since this problem is linked to your usage in SPDK, please test against -rc2 and confirm this is ok. > > In commit e072d16f8920 ("vfio: fix expanding DMA area in ppc64le"), I add= ed > a sanity check using a mapped address to resolve an issue around expandin= g > IOMMU window, but this was not enough, since memory allocation can return > memory anywhere dependent on memory fragmentation. DPDK may still skip DM= A > mapping and attempts to unmap non-mapped DMA during expanding IOMMU windo= w. > As a result, SPDK apps using large physical memory frequently failed to > proceed the communication with NVMe and/or went into an infinite loop. > > The root cause of the bug was in a gap between memory segments managed by > DPDK and firmware-level DMA mapping. DPDK's memory segments don't contain > the state of DMA mapping, and so, the memesg_walk cannot determine if an > iterated memory segment is mapped or not. This resulted in incorrect DMA > maps and unmaps. > > At this time, I added the code to avoid iterating non-mapped memory > segments during DMA mapping. The memseg_walk iterates over memory segment= s > marked as "used", and so, the code sets memory segments that will be > mapped or unmapped as "free" transiently. > > The commit db90b4969e2e ("vfio: retry creating sPAPR DMA window") allows > retring different page levels and sizes to create DMA window. However, th= is > allows page sizes different from hugepage sizes. This inconsistency cause= d > failures at the time of DMA mapping after the window creation. This patch > fixes to retry only different page levels. > > Fixes: e072d16f8920 ("vfio: fix expanding DMA area in ppc64le") > Fixes: db90b4969e2e ("vfio: retry creating sPAPR DMA window") Cc: stable@dpdk.org > > Signed-off-by: Takeshi Yoshimura Reviewed-by: David Christensen Applied, thanks. -- David Marchand