From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 88F3BA04DB for ; Thu, 15 Oct 2020 08:09:39 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 32E7D1DC88; Thu, 15 Oct 2020 08:09:38 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id 0EE151DA84; Thu, 15 Oct 2020 08:09:33 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09F65Krt005316; Wed, 14 Oct 2020 23:09:32 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=date : from : to : cc : subject : message-id : references : content-type : in-reply-to : mime-version; s=pfpt0220; bh=OZDKszxkKz2vEeoT4o4i8Ars15M5VWWUXGElaB6LHQI=; b=KlQPGzvlGGoxlc+0hnPmOp0PjndFwlDz67875xqeC4Ni5UIjD6edB4s6SqCLzwGrvtWo b3bsDDbJZeaOTTL1rfutOLJ8+NkXy/z559BGSVfUWXOqI/110Zt5IootCfT3NmlcXNUQ LgWbPJ0uyh2CEdZEgxKJYx8IIg9SjMKoDPCRqY6q1nF2gaqbgJ3WP5ri1oBirGuXInOM iuY6ZxnccwcivjLmXH8PlxZMVhi+0qqwdjvv9SjjFW0qNzjHLpotxQfzh9aJB43ZF3UL Na7YFR8M1ltv2EdG6d1pzbuGvJ2x8DK6hRaYNzIPqwmI7W83j4ex/ijYXNwJtMNmpOkN Fg== Received: from sc-exch01.marvell.com ([199.233.58.181]) by mx0a-0016f401.pphosted.com with ESMTP id 343aantjn1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 14 Oct 2020 23:09:32 -0700 Received: from SC-EXCH03.marvell.com (10.93.176.83) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 14 Oct 2020 23:09:31 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.172) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Wed, 14 Oct 2020 23:09:31 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cx/otiwyNErUKMTd7VDpYl2nz9tZhgR/vQ5gqAWznHEveedT3YWo+HZs3pPTSd97UTqe4LnFqIIOSM46RSf4Z+XOd3veV50ZcsvcIUEnE2sO6n/QNxttyDkseMgJ+VnB4WzqCe/XF8+IwCVBF6oP4s43HFyUHkFgi7d3LrB6ABhtV4/rPXVTfKsVHn2ATfElQxMjhbjaMP9ccRARHqCS4XaLhr8Mp3XlWBAD7D/0i5HuO6yagdeEmfRf4w/rzJdSnKW7VpIuwJF9L+8Wa4vd3Q0ht/u7DXNhGLEwLl5lMPWk5IvjQpLBDRLhiX9kWLEuW16ueJ+sc4efGJgoB4C0Rg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OZDKszxkKz2vEeoT4o4i8Ars15M5VWWUXGElaB6LHQI=; b=ONBnmxS8lMAYEySBpe9FWKY+tr6VQIX1xA9fF6qwLHnVmphcYN7J43u1mTQdBzl4HpOLsdZvz26fAWxPwG3OTl6aKDrVE2oxqNnu0+KgoQMcf9xqUm2XRpn0FF/1c2GpGpcma4dk4DztcwgIOjcrXhgIDUThf5ETCl08KvxiaiP1girykgq5UqDDlxufaljvXruspfE9QOh3s4CNvzi7kl/iYd+hiFZHclaJZBUVVBo0WAQg8wHfv0PW8q6Zt4004zxHCg7CLIV5sfK0Yk7o1uZKqeefNGgYl802sK9WbOkUCa3Rj1RXOghkx+znXLANsWJR74nvmUV1/uDLexYqKA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OZDKszxkKz2vEeoT4o4i8Ars15M5VWWUXGElaB6LHQI=; b=m8mjGFjqsv6q3t+sNfwu+iQnbSS225ot06uncOvUqg+mBrLJO0CZDPhG4a9v8SSakJMDEbbiS79c7Mvrb9LelXDP28WcGgUCpFt7bYmvgcML9ov66Whc5XeSYfrLvPLujJDPBJU3i7EWXMBC1y+ZW5NaUk9DeHhJC16cSEld5RY= Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=marvell.com; Received: from BN8PR18MB2386.namprd18.prod.outlook.com (2603:10b6:408:68::25) by BN6PR18MB1412.namprd18.prod.outlook.com (2603:10b6:404:f4::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3455.22; Thu, 15 Oct 2020 06:09:30 +0000 Received: from BN8PR18MB2386.namprd18.prod.outlook.com ([fe80::25af:7baf:528d:1dab]) by BN8PR18MB2386.namprd18.prod.outlook.com ([fe80::25af:7baf:528d:1dab%5]) with mapi id 15.20.3455.035; Thu, 15 Oct 2020 06:09:29 +0000 Date: Thu, 15 Oct 2020 11:39:14 +0530 From: Nithin Dabilpuram To: "Burakov, Anatoly" CC: , , Message-ID: <20201015060914.GA32207@outlook.office365.com> References: <20201012081106.10610-1-ndabilpuram@marvell.com> <20201012081106.10610-3-ndabilpuram@marvell.com> <05afb7f5-96bf-dffd-15dd-2024586f7290@intel.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <05afb7f5-96bf-dffd-15dd-2024586f7290@intel.com> User-Agent: Mutt/1.12.2 (34cd43c) (2019-09-21) X-Originating-IP: [1.6.215.26] X-ClientProxiedBy: BMXPR01CA0078.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:54::18) To BN8PR18MB2386.namprd18.prod.outlook.com (2603:10b6:408:68::25) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from outlook.office365.com (1.6.215.26) by BMXPR01CA0078.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:54::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3477.20 via Frontend Transport; Thu, 15 Oct 2020 06:09:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e5a15c9a-75fd-4580-407e-08d870d0e046 X-MS-TrafficTypeDiagnostic: BN6PR18MB1412: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:758; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SYyYX1zMKM4BcHDH5MbyCTP06KCWcZk2OZerPfFGLJ12UeroLXT3c+wTKu3OuAXsX2N8YnXbCIu23vaOIj04r0nRMim4kDbwyavMbQLYQ4q1S7V5/16AJMhRomQe61nUzHSRjOy64nBMkth02JeTDlZ49cfCjtYh3aI5KRlHRnw3gwFQLchLj/m33TXP49QWTEHie0Aq41YH+yuqudTAMd0bevNmNPSN9cn+PkzvhV2UCr0Qk8lGinJltjv7gggET6EK6sNFZ1JbBY8UnVd07/5k9bJzH8+4t0aNd6hyB1IKchH77TlRN3kDoCatgy8F8yAPQlw6WNIl7908qss0/cx8SFCCuhV9EZr07JkcVCUX+Pgy9+Chdxo5HkOBbdfGi0wyo/52YxYuo6CYBDqncsb5L0HVo5S57KxG48tIP3/EdNM9Q0xYkw/TGCzyzBQaJzk49tJXdsBqaL9QmAUoAJzVrWyJwN31xar/Bi44pxJ2LQfcgBf/2gkR6tAAwpU9 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN8PR18MB2386.namprd18.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(39860400002)(376002)(346002)(366004)(956004)(52116002)(2906002)(7696005)(9686003)(8676002)(316002)(83080400001)(966005)(26005)(6916009)(55016002)(1076003)(83380400001)(186003)(1006002)(16526019)(6666004)(66476007)(8936002)(5660300002)(4326008)(33656002)(66946007)(6506007)(53546011)(66556008)(86362001)(478600001)(34490700002)(36456003)(42976004); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: zva0dzYcG4M/kZf7+BpJdh4/XPlHRjJ/iBtsdfhDBU1aGdui5pFGcPj9umdnW7SbZAtIazDYfqAXAqufJQtfeSYu+maJpbepV2442+tbEqcwVTL64QlpElwAcbyIaxVqntnjdPmQ6BIt8mpcCnxTv/bywZBUHWeqEk8XXd46vcmMN4sO8l489WwleTV1Ku8fAoezWB/Yisu8ABGrbvpnE7EBtbHEFIbhGI/OSwfFwrlN3URTmtcT2f//thDEBNdKlUczh/VL8rNLf3pjMkHqTR0PZa4eDu63XH6TkwoyRt/3jrYTPdD2NjgQHnMjduidyxNkry4mhI7/L/Pz6o59zzOd9y1UvuG/P7ZgnWox+bqFMuIEdyPG5xyps8N5JA8hYJ04QZm+DenVwwQYkVjiWpfOzGOpkOKG0UFZyF7yb790LQ57FzliEL5mBRalgyGwYPYLkbQ7rVN4hK7xd8A32rRHItECo48NUqO1HUm7LepSyPzwosd0io5GEPIISZVsN5ZKmLGgPZAh7RraCgZuWiFK0jyM9DFZ8AXKLTOyVfc0+nh3CZxoORY6AI9e4txbFq0vI0qQ0c/l92PXpjWZ1PGZjJmRLnqAb8pStmyLlRlK2XhUHBd1nOattgrX7zNHvQ41xM3pw+nUy/kazRaM6A== X-MS-Exchange-CrossTenant-Network-Message-Id: e5a15c9a-75fd-4580-407e-08d870d0e046 X-MS-Exchange-CrossTenant-AuthSource: BN8PR18MB2386.namprd18.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Oct 2020 06:09:29.3312 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6wfcZFz3fYiVJk01NkkGreELwYhbQeKlqcocYILE9auodcyiodqsCpAsZciD+vgFx0QtCWJNuTM0M64N/0Ni4w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR18MB1412 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-10-15_03:2020-10-14, 2020-10-15 signatures=0 Subject: Re: [dpdk-stable] [EXT] Re: [PATCH 2/2] vfio: fix partial DMA unmapping for VFIO type1 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" On Wed, Oct 14, 2020 at 04:07:10PM +0100, Burakov, Anatoly wrote: > External Email > > ---------------------------------------------------------------------- > On 12-Oct-20 9:11 AM, Nithin Dabilpuram wrote: > > Partial unmapping is not supported for VFIO IOMMU type1 > > by kernel. Though kernel gives return as zero, the unmapped size > > returned will not be same as expected. So check for > > returned unmap size and return error. > > > > For case of DMA map/unmap triggered by heap allocations, > > maintain granularity of memseg page size so that heap > > expansion and contraction does not have this issue. > > This is quite unfortunate, because there was a different bug that had to do > with kernel having a very limited number of mappings available [1], as a > result of which the page concatenation code was added. > > It should therefore be documented that the dma_entry_limit parameter should > be adjusted should the user run out of the DMA entries. > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_lkml_155414977872.12780.13728555131525362206.stgit-40gimli.home_T_&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=3GMg-634_cdUCY4WpQPwjzZ_S4ckuMHOnt2FxyyjXMk&s=TJLzppkaDS95VGyRHX2hzflQfb9XLK0OiOszSXoeXKk&e= Ack, I'll document it in guides/linux_gsg/linux_drivers.rst in vfio section. > > > > > For user requested DMA map/unmap disallow partial unmapping > > for VFIO type1. > > > > Fixes: 73a639085938 ("vfio: allow to map other memory regions") > > Cc: anatoly.burakov@intel.com > > Cc: stable@dpdk.org > > > > Signed-off-by: Nithin Dabilpuram > > --- > > lib/librte_eal/linux/eal_vfio.c | 34 ++++++++++++++++++++++++++++------ > > lib/librte_eal/linux/eal_vfio.h | 1 + > > 2 files changed, 29 insertions(+), 6 deletions(-) > > > > diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c > > index d26e164..ef95259 100644 > > --- a/lib/librte_eal/linux/eal_vfio.c > > +++ b/lib/librte_eal/linux/eal_vfio.c > > @@ -69,6 +69,7 @@ static const struct vfio_iommu_type iommu_types[] = { > > { > > .type_id = RTE_VFIO_TYPE1, > > .name = "Type 1", > > + .partial_unmap = false, > > .dma_map_func = &vfio_type1_dma_map, > > .dma_user_map_func = &vfio_type1_dma_mem_map > > }, > > @@ -76,6 +77,7 @@ static const struct vfio_iommu_type iommu_types[] = { > > { > > .type_id = RTE_VFIO_SPAPR, > > .name = "sPAPR", > > + .partial_unmap = true, > > .dma_map_func = &vfio_spapr_dma_map, > > .dma_user_map_func = &vfio_spapr_dma_mem_map > > }, > > @@ -83,6 +85,7 @@ static const struct vfio_iommu_type iommu_types[] = { > > { > > .type_id = RTE_VFIO_NOIOMMU, > > .name = "No-IOMMU", > > + .partial_unmap = true, > > .dma_map_func = &vfio_noiommu_dma_map, > > .dma_user_map_func = &vfio_noiommu_dma_mem_map > > }, > > @@ -525,12 +528,19 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len, > > /* for IOVA as VA mode, no need to care for IOVA addresses */ > > if (rte_eal_iova_mode() == RTE_IOVA_VA && msl->external == 0) { > > uint64_t vfio_va = (uint64_t)(uintptr_t)addr; > > - if (type == RTE_MEM_EVENT_ALLOC) > > - vfio_dma_mem_map(default_vfio_cfg, vfio_va, vfio_va, > > - len, 1); > > - else > > - vfio_dma_mem_map(default_vfio_cfg, vfio_va, vfio_va, > > - len, 0); > > + uint64_t page_sz = msl->page_sz; > > + > > + /* Maintain granularity of DMA map/unmap to memseg size */ > > + for (; cur_len < len; cur_len += page_sz) { > > + if (type == RTE_MEM_EVENT_ALLOC) > > + vfio_dma_mem_map(default_vfio_cfg, vfio_va, > > + vfio_va, page_sz, 1); > > + else > > + vfio_dma_mem_map(default_vfio_cfg, vfio_va, > > + vfio_va, page_sz, 0); > > + vfio_va += page_sz; > > + } > > + > > You'd also have to revert d1c7c0cdf7bac5eb40d3a2a690453aefeee5887b because > currently the PA path will opportunistically concantenate contiguous > segments into single mapping too. Ack, I'll change it even for IOVA as PA mode. I missed that. > > > return; > > } > > @@ -1383,6 +1393,12 @@ vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova, > > RTE_LOG(ERR, EAL, " cannot clear DMA remapping, error %i (%s)\n", > > errno, strerror(errno)); > > return -1; > > + } else if (dma_unmap.size != len) { > > + RTE_LOG(ERR, EAL, " unexpected size %"PRIu64" of DMA " > > + "remapping cleared instead of %"PRIu64"\n", > > + (uint64_t)dma_unmap.size, len); > > + rte_errno = EIO; > > + return -1; > > } > > } > > @@ -1853,6 +1869,12 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova, > > /* we're partially unmapping a previously mapped region, so we > > * need to split entry into two. > > */ > > + if (!vfio_cfg->vfio_iommu_type->partial_unmap) { > > + RTE_LOG(DEBUG, EAL, "DMA partial unmap unsupported\n"); > > + rte_errno = ENOTSUP; > > + ret = -1; > > + goto out; > > + } > > How would we ever arrive here if we never do more than 1 page worth of > memory anyway? I don't think this is needed. container_dma_unmap() is called by user via rte_vfio_container_dma_unmap() and when he maps we don't split it as we don't about his memory. So if he maps multiple pages and tries to unmap partially, then we should fail. > > > if (user_mem_maps->n_maps == VFIO_MAX_USER_MEM_MAPS) { > > RTE_LOG(ERR, EAL, "Not enough space to store partial mapping\n"); > > rte_errno = ENOMEM; > > diff --git a/lib/librte_eal/linux/eal_vfio.h b/lib/librte_eal/linux/eal_vfio.h > > index cb2d35f..6ebaca6 100644 > > --- a/lib/librte_eal/linux/eal_vfio.h > > +++ b/lib/librte_eal/linux/eal_vfio.h > > @@ -113,6 +113,7 @@ typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova, > > struct vfio_iommu_type { > > int type_id; > > const char *name; > > + bool partial_unmap; > > vfio_dma_user_func_t dma_user_map_func; > > vfio_dma_func_t dma_map_func; > > }; > > > > > -- > Thanks, > Anatoly