From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <alejandro.lucero@netronome.com> Received: from mail-vk0-f46.google.com (mail-vk0-f46.google.com [209.85.213.46]) by dpdk.org (Postfix) with ESMTP id BAE8069C5 for <dev@dpdk.org>; Mon, 13 Feb 2017 14:38:25 +0100 (CET) Received: by mail-vk0-f46.google.com with SMTP id x75so60422268vke.2 for <dev@dpdk.org>; Mon, 13 Feb 2017 05:38:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=igOhEAn+3Q+TGl9v0I+durtw/t6orxWgIGzslp2rDBQ=; b=SB0eQwb3jGmyfm6hP1PvRixS+v+VtXnlprfqVjYvCfmkmCkLWu4ciQPjDN7wcfEeoj IcLCcTXJJQShKyn5veFAGbpstWju7cpVTUPFyBLx9K529F+w04NoSbG2/oYnHxYQE048 j+lax527dKl1awuTl6oAO8FdFwbYkNnWgnjEJKvPor8i3FgqcbGHtzZlHe9HuEX3KRYy jRXUSJz+ohWYKi90ob3QjCpgTVrh4zcSZINJs0oANHzH+tzITOeiRsqNuzSqpkk5abVP OIHQT2BEIIJAPcZu3nfrq8kgC59CWs9CNe378eqT38qkAKuCQ1D4uYRLhlWqial1wekN XFAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=igOhEAn+3Q+TGl9v0I+durtw/t6orxWgIGzslp2rDBQ=; b=mPKrJydd2HoJBraJjFN+72M9G5vEqXLa7e+QE51sDTW0ZDDnJVbWqfkBnBmGH8ncu9 Pmemfsik+Ue8YEI3arRGrlj9SKCCuYLskmobHId1OLaKll9EqLHleFMFxZQxFXIppuXg tO4at8AqVRaNRGc3sQEwb3g1BuRiZKfVW+hffO+bF8hhRq1O3eELMVpcP5priClasjLB wFsRhhb0wHalCkGEJdOUx7XzCAiNZZUKixjIAANv67HUXLs/CdeoNvFtc7y1SQtcHNIE +hDV20NIKASR/kF2BGkPK02SqOfDV5P9f5mFW/nhCBSZ9G6ddirKdQ7CuYG3DRebnjKb JCog== X-Gm-Message-State: AMke39lixLNHkqSxhlP1sgUAr+86Wpjp3ePhEPmwrwghllu9J+PKAn260LiBIBiPnSfkAXCdXMVvCKKo83OJrqWg X-Received: by 10.31.204.197 with SMTP id c188mr11289522vkg.31.1486993104880; Mon, 13 Feb 2017 05:38:24 -0800 (PST) MIME-Version: 1.0 Received: by 10.103.146.141 with HTTP; Mon, 13 Feb 2017 05:38:24 -0800 (PST) In-Reply-To: <6f31fa21-8de8-961d-e66c-7824c65ab5fc@intel.com> References: <1484742475-41005-1-git-send-email-alejandro.lucero@netronome.com> <67a9fd3b-b7f5-c641-9f59-590155cbd30b@intel.com> <CAD+H993AnUWkwpoBQf+nXpkHEXL6D_igkt3y4RAx4ixJ+47cjQ@mail.gmail.com> <d499b34f-7a13-bcdb-cf26-bba5a0ada247@intel.com> <6f31fa21-8de8-961d-e66c-7824c65ab5fc@intel.com> From: Alejandro Lucero <alejandro.lucero@netronome.com> Date: Mon, 13 Feb 2017 13:38:24 +0000 Message-ID: <CAD+H993F-RhUi1oCuOwEYmej75MD_ZSBWpq3aV_eDmY_Y+6gEw@mail.gmail.com> To: Ferruh Yigit <ferruh.yigit@intel.com> Cc: dev <dev@dpdk.org> Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Mon, 13 Feb 2017 13:38:26 -0000 On Fri, Feb 10, 2017 at 7:06 PM, Ferruh Yigit <ferruh.yigit@intel.com> wrote: > On 2/10/2017 7:03 PM, Ferruh Yigit wrote: > > On 2/8/2017 11:54 AM, Alejandro Lucero wrote: > >> Hi Ferruh, > >> > >> On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit <ferruh.yigit@intel.com > >> <mailto:ferruh.yigit@intel.com>> wrote: > >> > >> Hi Alejandro, > >> > >> On 1/18/2017 12:27 PM, Alejandro Lucero wrote: > >> > For using a DPDK app when iommu is enabled, it requires to > >> > add iommu=pt to the kernel command line. But using igb_uio driver > >> > makes DMAR errors because the device has not an IOMMU domain. > >> > >> Please help to understand the scope of the problem, > >> > >> > >> After reading your reply, I realize I could have explained it better. > >> First of all, this is related to SRIOV, exactly when the VFs are > created. > >> > >> > >> 1- How can you re-produce the problem? > >> > >> > >> Using a VF from a Intel card by a DPDK app in the host and a kernel >= > >> 3.15. Although usually VFs are assigned to VMs, it could also be an > >> option to use VFs by the host. > >> > >> BTW, I did not try to reproduce the problem with an Intel card. I > >> triggered this problem with an NFP, but because the problem behind, I > >> bet that is going to happen for an Intel one as well. > > > > I can able to reproduce the problem with ixgbe, by using VF on the host. > > > > And I verified your patch fixes it, it cause device attached to a vfio > > group. > > I want to send this in a separate mail, since not directly related to > your patch, but while I was testing with vfio-pci I get lower numbers > comparing to the igb_uio, which is unexpected AFAIK. > > Most probably I am doing something wrong, but I would like to ask if are > you observing same behavior? > Can you tell me which test are you running? Although both, igb_uio and vfio, allow to work with IOMMU, the first one requires iommu=pt. It implies a single IOMMU domain already created by the system with the 1:1 mapping being used. With VFIO, a specific per device IOMMU domain is created. Depending on how are you measuring performance, that specific IOMMU domain creation by the DPDK app could have an impact, but I do not think that should be really significant. But with IOMMU you have the same problem than with MMU, there is a IOTLB for IOMMU as there is a TLB for MMU. Depending on the app, some IOMMU/IOTLB contention is likely. I have done some experiments and still investigating this during my spare time. It would be worth a talk about this in the next DPDK meeting. > > Thanks, > ferruh > >