From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 44759A0532; Wed, 22 Jan 2020 19:35:02 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 29E872C6D; Wed, 22 Jan 2020 19:35:01 +0100 (CET) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by dpdk.org (Postfix) with ESMTP id 83C782BF2 for ; Wed, 22 Jan 2020 19:34:59 +0100 (CET) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 00MISBjg094419 for ; Wed, 22 Jan 2020 13:34:58 -0500 Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0a-001b2d01.pphosted.com with ESMTP id 2xp945bnuw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 22 Jan 2020 13:34:58 -0500 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 00MIUO4f010215 for ; Wed, 22 Jan 2020 18:34:57 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma02wdc.us.ibm.com with ESMTP id 2xksn6qgeg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 22 Jan 2020 18:34:57 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 00MIYsLp51118416 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 22 Jan 2020 18:34:54 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6AA7B2805C; Wed, 22 Jan 2020 18:34:54 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2581D28058; Wed, 22 Jan 2020 18:34:54 +0000 (GMT) Received: from davids-mbp.usor.ibm.com (unknown [9.70.85.94]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 22 Jan 2020 18:34:54 +0000 (GMT) To: Takeshi Yoshimura , dev@dpdk.org Cc: drc@ibm.com References: <20200117042555.22567-1-tyos@jp.ibm.com> From: David Christensen Message-ID: <024a05ec-039a-e3ee-fa06-9e029805a32c@linux.vnet.ibm.com> Date: Wed, 22 Jan 2020 10:34:53 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: <20200117042555.22567-1-tyos@jp.ibm.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-01-22_08:2020-01-22, 2020-01-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 adultscore=0 impostorscore=0 mlxscore=0 priorityscore=1501 bulkscore=0 lowpriorityscore=0 mlxlogscore=999 suspectscore=2 clxscore=1015 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1910280000 definitions=main-2001220156 Subject: Re: [dpdk-dev] [PATCH] vfio: fix VFIO mapping failures in ppc64le X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > ppc64le failed when using large physical memory. I found problems in my two > commits in the past. > > In commit e072d16f8920 ("vfio: fix expanding DMA area in ppc64le"), I added > a sanity check using a mapped address to resolve an issue around expanding > IOMMU window, but this was not enough, since memory allocation can return > memory anywhere dependent on memory fragmentation. DPDK may still skip DMA > mapping and attempts to unmap non-mapped DMA during expanding IOMMU window. > As a result, SPDK apps using large physical memory frequently failed to > proceed the communication with NVMe and/or went into an infinite loop. > > The root cause of the bug was in a gap between memory segments managed by > DPDK and firmware-level DMA mapping. DPDK's memory segments don't contain > the state of DMA mapping, and so, the memesg_walk cannot determine if an > iterated memory segment is mapped or not. This resulted in incorrect DMA > maps and unmaps. > > At this time, I added the code to avoid iterating non-mapped memory > segments during DMA mapping. The memseg_walk iterates over memory segments > marked as "used", and so, the code sets memory segments that will be > mapped or unmapped as "free" transiently. > > The commit db90b4969e2e ("vfio: retry creating sPAPR DMA window") allows > retring different page levels and sizes to create DMA window. However, this > allows page sizes different from hugepage sizes. This inconsistency caused > failures at the time of DMA mapping after the window creation. This patch > fixes to retry only different page levels. > > Fixes: e072d16f8920 ("vfio: fix expanding DMA area in ppc64le") > Fixes: db90b4969e2e ("vfio: retry creating sPAPR DMA window") > > Signed-off-by: Takeshi Yoshimura > --- Reviewed-by: David Christensen