From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <shivpri.b@gmail.com> Received: from mail-ig0-f169.google.com (mail-ig0-f169.google.com [209.85.213.169]) by dpdk.org (Postfix) with ESMTP id 0CCA35906 for <dev@dpdk.org>; Mon, 27 Oct 2014 18:19:30 +0100 (CET) Received: by mail-ig0-f169.google.com with SMTP id a13so917372igq.4 for <dev@dpdk.org>; Mon, 27 Oct 2014 10:28:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=fBNx9wAksprfCLJPYR5dbxvJswCkDYafXe5AmIdSWiU=; b=LXnBNZTkyPyH9vD/fQuKhNtX33TJ3soyLfpu97zwAzc5xV+Ufsz3ZXnu48W7zmU/Xa tQiwSFV4GplRFawP+SS+SIHQaltDXmlvtIesjs7XL/704M4VMLhsWDfKIM/iEpXLdWZY PTV1/hVOHOh7cQlQuaYllXOF5oGMvZ4axunPYJXQ8IqpKd+2lupOwuHJBdkIVUvTxGMK dCETXQ5Xnfpy7BJWeg/9NJWHuTPr0S3fzuAsLASMRPyN4+JDuLcUCbFw+GCWKwB1zPUL L83pSgNuF+BZaK5TjbnGfykuf560QU1MIp2JjuYWzsqhEfp0kiNrwUmd8+/rIGpY0t6T dH9g== X-Received: by 10.107.11.80 with SMTP id v77mr3808547ioi.76.1414430893027; Mon, 27 Oct 2014 10:28:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.171.10 with HTTP; Mon, 27 Oct 2014 10:27:51 -0700 (PDT) In-Reply-To: <DFDF335405C17848924A094BC35766CF0A97AF0A@SHSMSX104.ccr.corp.intel.com> References: <CAKU5Fo_7fQHcYa-1ZBX8=D0SCkxZL8VLEpWYtJ6rG4DpV=g3gw@mail.gmail.com> <C37D651A908B024F974696C65296B57B0F2CD832@SHSMSX101.ccr.corp.intel.com> <CAKfHP0U-v2fLQwyEa3KBZs3kXpVMQdLU+WgLzQq6xxmrb3Brtg@mail.gmail.com> <CAKU5Fo-djWmG46gCYUScqOwJzDFoAwPvJvUrBdwss+D=nHLVqA@mail.gmail.com> <DFDF335405C17848924A094BC35766CF0A97795F@SHSMSX104.ccr.corp.intel.com> <CAKfHP0WstpCYb0POKP_bOt36+qfcAGj-j=ij9BeCZifRjJ080g@mail.gmail.com> <D0158A423229094DA7ABF71CF2FA0DA311851099@shsmsx102.ccr.corp.intel.com> <DFDF335405C17848924A094BC35766CF0A97AF0A@SHSMSX104.ccr.corp.intel.com> From: Shivapriya Hiremath <shivpri.b@gmail.com> Date: Mon, 27 Oct 2014 10:27:51 -0700 Message-ID: <CAKU5Fo8JrDmRsXLfaXzR2xkF2o5jRqRpgCAE3L9HeNGM3Krcyw@mail.gmail.com> To: "Zhou, Danny" <danny.zhou@intel.com> Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" <dev@dpdk.org> Subject: Re: [dpdk-dev] Why do we need iommu=pt? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Mon, 27 Oct 2014 17:19:30 -0000 Hi Danny, Your reply was very helpful in understanding the impact. Can you please tell us if you saw any performance impact to DPDK when iommu=on ? -Shivapriya On Wed, Oct 22, 2014 at 8:21 AM, Zhou, Danny <danny.zhou@intel.com> wrote: > Echo Cunming and we did not see obvious performance impact when iommu = pt > is used despite of > igb_uio or VFIO is used. > > Alex, > The map and umap operation for each e/ingress packet is done by hw rather > than sw, so > performance impact to DPDK should be minimum in my mind. If it actually > impacst perf, say on 100G NIC, > I am sure it will be resolved in next generation Intel silicon. We will be > performing some performance > tests with iommu = on to see any performance degradation. I cannot share > the detailed performance > result here on the community, but I could tell if it really bring negative > performance impact to DPDK. > Please stay tuned. > > Alex, > > > -----Original Message----- > > From: Liang, Cunming > > Sent: Wednesday, October 22, 2014 4:53 PM > > To: alex; Zhou, Danny > > Cc: dev@dpdk.org > > Subject: RE: [dpdk-dev] Why do we need iommu=pt? > > > > I thinks it's a good point using dma_addr rather than phys_addr. > > Without iommu, the value of them are the same. > > With iommu, the dma_addr value equal to the iova. > > It's not all for DPDK working with iommu but not pass through. > > > > We know each iova belongs to one iommu domain. > > And each device can attach to one domain. > > It means the iova will have coupling relationship with domain/device. > > > > Looking back to DPDK descriptor ring, it's all right, already coupling > with device. > > But if for mbuf mempool, in most cases, it's shared by multiple ports. > > So if keeping the way, all those ports/device need to put into the same > iommu domain. > > And the mempool has attach to specific domain, but not just the device. > > On this time, iommu domain no longer be transparent in DPDK. > > Vfio provide the verbs to control domain, we still need library to > manager such domain with mempool. > > > > All that overhead just make DPDK works with iommu in host, but remember > pt always works. > > The isolation of devices mainly for security concern. > > If it's not necessary, pt definitely is a good choice without > performance impact. > > > > For those self-implemented PMD using the DMA kernel interface to set up > its mappings appropriately. > > It don't require "iommu=pt". The default option "iommu=on" also works. > > > > > -----Original Message----- > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of alex > > > Sent: Wednesday, October 22, 2014 3:36 PM > > > To: Zhou, Danny > > > Cc: dev@dpdk.org > > > Subject: Re: [dpdk-dev] Why do we need iommu=pt? > > > > > > Shiva. > > > The cost of disabling iommu=pt when intel_iommu=on is dire. DPDK won't > work > > > as the RX/TX descriptors will be useless. > > > Any dam access by the device will be dropped as no dam-mapping will > exists. > > > > > > Danny. > > > The IOMMU hurts performance in kernel drivers which perform a map and > umap > > > operation for each e/ingress packet. > > > The costs of unmapping when under strict protection limit a +10Gb to > 3Gb > > > with cpu maxed out at 100%. DPDK apps shouldn't feel any difference > IFF the > > > rx descriptors contain iova and not real physical addresses which are > used > > > currently. > > > > > > > > > On Tue, Oct 21, 2014 at 10:10 PM, Zhou, Danny <danny.zhou@intel.com> > wrote: > > > > > > > IMHO, if memory protection with IOMMU is needed or not really > depends on > > > > how you use > > > > and deploy your DPDK based applications. For Telco network middle > boxes, > > > > which adopts > > > > a "close model" solution to achieve extremely high performance, the > entire > > > > system including > > > > HW, software in kernel and userspace are controlled by Telco vendors > and > > > > assumed trustable, so > > > > memory protection is not so important. While for Datacenters, which > > > > generally adopts a "open model" > > > > solution allows running user space applications(e.g. tenant > applications > > > > and VMs) which could > > > > direct access NIC and DMA engine inside the NIC using modified DPDK > PMD > > > > are not trustable > > > > as they can potentially DAM to/from arbitrary memory regions using > > > > physical addresses, so IOMMU > > > > is needed to provide strict memory protection, at the cost of > negative > > > > performance impact. > > > > > > > > So if you want to seek high performance, disable IOMMU in BIOS or > OS. And > > > > if security is a major > > > > concern, tune it on and tradeoff between performance and security. > But I > > > > do NOT think is comes with > > > > an extremely high performance costs according to our performance > > > > measurement, but it probably true > > > > for 100G NIC. > > > > > > > > > -----Original Message----- > > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shivapriya > Hiremath > > > > > Sent: Wednesday, October 22, 2014 12:54 AM > > > > > To: Alex Markuze > > > > > Cc: dev@dpdk.org > > > > > Subject: Re: [dpdk-dev] Why do we need iommu=pt? > > > > > > > > > > Hi, > > > > > > > > > > Thank you for all the replies. > > > > > I am trying to understand the impact of this on DPDK. What will be > the > > > > > repercussions of disabling "iommu=pt" on the DPDK performance? > > > > > > > > > > > > > > > On Tue, Oct 21, 2014 at 12:32 AM, Alex Markuze <alex@weka.io> > wrote: > > > > > > > > > > > DPDK uses a 1:1 mapping and doesn't support IOMMU. IOMMU allows > for > > > > > > simpler VM physical address translation. > > > > > > The second role of IOMMU is to allow protection from unwanted > memory > > > > > > access by an unsafe devise that has DMA privileges. > Unfortunately this > > > > > > protection comes with an extremely high performance costs for > high > > > > speed > > > > > > nics. > > > > > > > > > > > > To your question iommu=pt disables IOMMU support for the > hypervisor. > > > > > > > > > > > > On Tue, Oct 21, 2014 at 1:39 AM, Xie, Huawei < > huawei.xie@intel.com> > > > > wrote: > > > > > > > > > > > >> > > > > > >> > > > > > >> > -----Original Message----- > > > > > >> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of > Shivapriya > > > > > >> Hiremath > > > > > >> > Sent: Monday, October 20, 2014 2:59 PM > > > > > >> > To: dev@dpdk.org > > > > > >> > Subject: [dpdk-dev] Why do we need iommu=pt? > > > > > >> > > > > > > >> > Hi, > > > > > >> > > > > > > >> > My question is that if the Poll mode driver used the DMA > kernel > > > > > >> interface > > > > > >> > to set up its mappings appropriately, would it still require > that > > > > > >> iommu=pt > > > > > >> > be set? > > > > > >> > What is the purpose of setting iommu=pt ? > > > > > >> PMD allocates memory though hugetlb file system, and fills the > > > > physical > > > > > >> address > > > > > >> into the descriptor. > > > > > >> pt is used to pass through iotlb translation. Refer to the > below link. > > > > > >> http://lkml.iu.edu/hypermail/linux/kernel/0906.2/02129.html > > > > > >> > > > > > > >> > Thank you. > > > > > >> > > > > > > > > > > > > > > > > >