From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id B67201B05 for ; Fri, 29 Mar 2019 13:40:17 +0100 (CET) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 5471721EDB; Fri, 29 Mar 2019 08:40:17 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 29 Mar 2019 08:40:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=Dz9F6boU8ueHfLMJwdNwSbNs2NditNO+ObrTR9jze20=; b=AotPCm7Qu4jJ hsYK7MoOLZGhiBlkKdhRKUrbGrfeog1Ddsmn+6uYUOwVhQwByyuqq+Q8N2inTv6i rlCcCJJu9QYi7VFaFq/QPamtJN1iSIrA7koJFezA7CLr9GkFfhDQDLMpX0CikrBB Q7AapgwWefaqKU20GNJsw0DZbpACJdk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=Dz9F6boU8ueHfLMJwdNwSbNs2NditNO+ObrTR9jze 20=; b=x66gNOKdxsO9F+U/vK0UdPy4uhaVwlzXbMTzV71mCm5L75IwHABiw+Daz /o5LC/FC8wEgemrGsksbHUZgWgLf1ewwiLJjJq4fi3JtIovnxqKWC3UYmMVHv/EI Ju+a9tUMWVgb1Zfv9Gs1iuxyV1q9PDhSKMrlqJBj3UK8ZrI1bDgGWkP1/BRkQ8FP UdwDIbwxXlenYPV/pvNx7vyGrg/XxLxuZCMjV6m6bOFZj2dq49l2eXRuj8dlOn7c l0Ow+4S+GAG8/dMRUo7oBXlkMrJnG7j4f5shFSkvayFfgSuElpL/5w1UYXwbXDm3 0Ea6CKLENHCmCFnDOkbSFOQCv9vdg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrkeejgdduiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucffoh hmrghinhepughpughkrdhorhhgnecukfhppeejjedrudefgedrvddtfedrudekgeenucfr rghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlhhonhdrnhgvthenuc evlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 97DD61031A; Fri, 29 Mar 2019 08:40:15 -0400 (EDT) From: Thomas Monjalon To: "Burakov, Anatoly" Cc: David Marchand , dev , John McNamara , Marko Kovacevic , iain.barker@oracle.com, edwin.leung@oracle.com, maxime.coquelin@redhat.com Date: Fri, 29 Mar 2019 13:40:14 +0100 Message-ID: <3255576.YcZt162MTL@xps> In-Reply-To: References: <07f664c33ddedaa5dcfe82ecb97d931e68b7e33a.1550855529.git.anatoly.burakov@intel.com> <1682850.JO3elT0QtZ@xps> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-dev] [PATCH] eal: add option to not store segment fd's X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2019 12:40:18 -0000 29/03/2019 13:05, Burakov, Anatoly: > On 29-Mar-19 11:34 AM, Thomas Monjalon wrote: > > 29/03/2019 11:33, Burakov, Anatoly: > >> On 29-Mar-19 9:50 AM, David Marchand wrote: > >>> On Fri, Feb 22, 2019 at 6:12 PM Anatoly Burakov > >>> > wrote: > >>> > >>> Due to internal glibc limitations [1], DPDK may exhaust internal > >>> file descriptor limits when using smaller page sizes, which results > >>> in inability to use system calls such as select() by user > >>> applications. > >>> > >>> While the problem can be worked around using --single-file-segments > >>> option, it does not work if --legacy-mem mode is also used. Add a > >>> (yet another) EAL flag to disable storing fd's internally. This > >>> will sacrifice compability with Virtio with vhost-backend, but > >>> at least select() and friends will work. > >>> > >>> [1] https://mails.dpdk.org/archives/dev/2019-February/124386.html > >>> > >>> > >>> Sorry, I am a bit lost and I never took the time to look in the new > >>> memory allocation system. > >>> This gives the impression that we are accumulating workarounds, between > >>> legacy-mem, single-file-segments, now no-seg-fds. > >> > >> Yep. I don't like this any more than you do, but i think there are users > >> of all of these, so we can't just drop them willy-nilly. My great hope > >> was that by now everyone would move on to use VFIO so legacy mem > >> wouldn't be needed (the only reason it exists is to provide > >> compatibility for use cases where lots of IOVA-contiguous memory is > >> required, and VFIO cannot be used), but apparently that is too much to > >> ask :/ > >> > >>> > >>> Iiuc, everything revolves around the need for per page locks. > >>> Can you summarize why we need them? > >> > >> The short answer is multiprocess. We have to be able to map and unmap > >> pages individually, and for that we need to be sure that we can, in > >> fact, remove a page because no one else uses it. We also need to store > >> fd's because virtio with vhost-user backend needs them to work, because > >> it relies on sharing memory between processes using fd's. > > > > It's a pity adding an option to workaround a limitation of a corner case. > > It adds complexity that we will have to support forever, > > and it's even not perfect because of vhost. > > > > Might there be another solution? > > > > If there is one, i'm all ears. I don't see any solutions aside from > adding limitations. > > For example, we could drop the single/multi file segments mode and just > make single file segments a default and the only available mode, but > this has certain risks because older kernels do not support fallocate() > on hugetlbfs. > > We could further draw a line in the sand, and say that, for example, > 19.11 (or 20.11) will not have legacy mem mode, and everyone should use > VFIO by now and if you don't it's your own fault. > > We could also cut down on the number of fd's we use in single-file > segments mode by not using locks and simply deleting pages in the > primary, but yanking out hugepages from under secondaries' feet makes me > feel uneasy, even if technically by the time that happens, they're not > supposed to be used anyway. This could mean that the patch is no longer > necessary because we don't use that many fd's any more. This last option is interesting. Is it realistic? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 5E263A05D3 for ; Fri, 29 Mar 2019 13:40:21 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 291762965; Fri, 29 Mar 2019 13:40:20 +0100 (CET) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id B67201B05 for ; Fri, 29 Mar 2019 13:40:17 +0100 (CET) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 5471721EDB; Fri, 29 Mar 2019 08:40:17 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 29 Mar 2019 08:40:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=Dz9F6boU8ueHfLMJwdNwSbNs2NditNO+ObrTR9jze20=; b=AotPCm7Qu4jJ hsYK7MoOLZGhiBlkKdhRKUrbGrfeog1Ddsmn+6uYUOwVhQwByyuqq+Q8N2inTv6i rlCcCJJu9QYi7VFaFq/QPamtJN1iSIrA7koJFezA7CLr9GkFfhDQDLMpX0CikrBB Q7AapgwWefaqKU20GNJsw0DZbpACJdk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=Dz9F6boU8ueHfLMJwdNwSbNs2NditNO+ObrTR9jze 20=; b=x66gNOKdxsO9F+U/vK0UdPy4uhaVwlzXbMTzV71mCm5L75IwHABiw+Daz /o5LC/FC8wEgemrGsksbHUZgWgLf1ewwiLJjJq4fi3JtIovnxqKWC3UYmMVHv/EI Ju+a9tUMWVgb1Zfv9Gs1iuxyV1q9PDhSKMrlqJBj3UK8ZrI1bDgGWkP1/BRkQ8FP UdwDIbwxXlenYPV/pvNx7vyGrg/XxLxuZCMjV6m6bOFZj2dq49l2eXRuj8dlOn7c l0Ow+4S+GAG8/dMRUo7oBXlkMrJnG7j4f5shFSkvayFfgSuElpL/5w1UYXwbXDm3 0Ea6CKLENHCmCFnDOkbSFOQCv9vdg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrkeejgdduiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucffoh hmrghinhepughpughkrdhorhhgnecukfhppeejjedrudefgedrvddtfedrudekgeenucfr rghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlhhonhdrnhgvthenuc evlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 97DD61031A; Fri, 29 Mar 2019 08:40:15 -0400 (EDT) From: Thomas Monjalon To: "Burakov, Anatoly" Cc: David Marchand , dev , John McNamara , Marko Kovacevic , iain.barker@oracle.com, edwin.leung@oracle.com, maxime.coquelin@redhat.com Date: Fri, 29 Mar 2019 13:40:14 +0100 Message-ID: <3255576.YcZt162MTL@xps> In-Reply-To: References: <07f664c33ddedaa5dcfe82ecb97d931e68b7e33a.1550855529.git.anatoly.burakov@intel.com> <1682850.JO3elT0QtZ@xps> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH] eal: add option to not store segment fd's X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190329124014.fGG7bMDHFmm52Be1O5-_V1dIxz67rX4nxH2Ox5e2d9o@z> 29/03/2019 13:05, Burakov, Anatoly: > On 29-Mar-19 11:34 AM, Thomas Monjalon wrote: > > 29/03/2019 11:33, Burakov, Anatoly: > >> On 29-Mar-19 9:50 AM, David Marchand wrote: > >>> On Fri, Feb 22, 2019 at 6:12 PM Anatoly Burakov > >>> > wrote: > >>> > >>> Due to internal glibc limitations [1], DPDK may exhaust internal > >>> file descriptor limits when using smaller page sizes, which results > >>> in inability to use system calls such as select() by user > >>> applications. > >>> > >>> While the problem can be worked around using --single-file-segments > >>> option, it does not work if --legacy-mem mode is also used. Add a > >>> (yet another) EAL flag to disable storing fd's internally. This > >>> will sacrifice compability with Virtio with vhost-backend, but > >>> at least select() and friends will work. > >>> > >>> [1] https://mails.dpdk.org/archives/dev/2019-February/124386.html > >>> > >>> > >>> Sorry, I am a bit lost and I never took the time to look in the new > >>> memory allocation system. > >>> This gives the impression that we are accumulating workarounds, between > >>> legacy-mem, single-file-segments, now no-seg-fds. > >> > >> Yep. I don't like this any more than you do, but i think there are users > >> of all of these, so we can't just drop them willy-nilly. My great hope > >> was that by now everyone would move on to use VFIO so legacy mem > >> wouldn't be needed (the only reason it exists is to provide > >> compatibility for use cases where lots of IOVA-contiguous memory is > >> required, and VFIO cannot be used), but apparently that is too much to > >> ask :/ > >> > >>> > >>> Iiuc, everything revolves around the need for per page locks. > >>> Can you summarize why we need them? > >> > >> The short answer is multiprocess. We have to be able to map and unmap > >> pages individually, and for that we need to be sure that we can, in > >> fact, remove a page because no one else uses it. We also need to store > >> fd's because virtio with vhost-user backend needs them to work, because > >> it relies on sharing memory between processes using fd's. > > > > It's a pity adding an option to workaround a limitation of a corner case. > > It adds complexity that we will have to support forever, > > and it's even not perfect because of vhost. > > > > Might there be another solution? > > > > If there is one, i'm all ears. I don't see any solutions aside from > adding limitations. > > For example, we could drop the single/multi file segments mode and just > make single file segments a default and the only available mode, but > this has certain risks because older kernels do not support fallocate() > on hugetlbfs. > > We could further draw a line in the sand, and say that, for example, > 19.11 (or 20.11) will not have legacy mem mode, and everyone should use > VFIO by now and if you don't it's your own fault. > > We could also cut down on the number of fd's we use in single-file > segments mode by not using locks and simply deleting pages in the > primary, but yanking out hugepages from under secondaries' feet makes me > feel uneasy, even if technically by the time that happens, they're not > supposed to be used anyway. This could mean that the patch is no longer > necessary because we don't use that many fd's any more. This last option is interesting. Is it realistic?