From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ua0-f172.google.com (mail-ua0-f172.google.com [209.85.217.172]) by dpdk.org (Postfix) with ESMTP id BB86DDE5 for ; Wed, 8 Feb 2017 12:54:44 +0100 (CET) Received: by mail-ua0-f172.google.com with SMTP id y9so107587614uae.2 for ; Wed, 08 Feb 2017 03:54:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=8eanU6bKoObZQSBnk3VCIpE3ggdELixPqmNjtl6+dds=; b=hbaCDxr/StjCbZw1EY0QYe0b1AIDy/Oul68wjNgp6uOebfrYQ9bMy9ALRFkcwalfs3 ssX+Cj+F2RjZMNcRhXe5UUR1cyYmWv/mnpxwLgAUDfjOwtvaKRyrnw0YmhVF0SRXPUcL dBkEiK+R7fEqrciZMEXg0vlgkXUK8FuJ196xybJONiqT31j8wvhDmTGfIvg3UlciN5Vk c4GBNCrneHYlyBtT6IffVQh78SJkZhEgJZXw5nm+uLPiI3xLmzNdDS7BCgiz4+k8ToCx eca8UM00GLmvDBJvh1G0PBED+Z/CKPzE/gNba9z6E4/Mmk5kAHxkQiitXhxEpaz8GWRo sajA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=8eanU6bKoObZQSBnk3VCIpE3ggdELixPqmNjtl6+dds=; b=bcSsJj0FRpa/dgSCrJf3Hx1b4m7vYd2CAWQVLhe9XOLEUhUQgDYg9so5LseJrIGiBW Xey+JiN7+h8df8DPxI3wYYSlI5HpYBBC+AwgaCFW7LiNpmMpZLlXqkHD/rjvfALFZBfB KQ1ERcDYo17VptxiJfXV+yOqNtzo4c6IoAG++I7Yup/6wli8Au3zlAu79vnqcJ9XRu+/ UwuBLZ887SVm+nwINzJfJnFaG5Da7HmVO6RttqH8eCtLmn+heNxwIQKHVhJZNUaqtucI NrVkns0wciNVyllStU9gAtQmwfmfLbDfxsSo8sP/GXRQzUXUkz7iK0Y04ERF0qLwrXhs z9kw== X-Gm-Message-State: AIkVDXLuBiUdMNQZ9yReO3ytxBeHUGWDPnhvxStV9I/l8duDxeLafkxgOeI5K2GfPPqY1KSMu0cPVjuRGa/5iP6L X-Received: by 10.159.39.72 with SMTP id a66mr10406015uaa.150.1486554883594; Wed, 08 Feb 2017 03:54:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.103.146.141 with HTTP; Wed, 8 Feb 2017 03:54:43 -0800 (PST) In-Reply-To: <67a9fd3b-b7f5-c641-9f59-590155cbd30b@intel.com> References: <1484742475-41005-1-git-send-email-alejandro.lucero@netronome.com> <67a9fd3b-b7f5-c641-9f59-590155cbd30b@intel.com> From: Alejandro Lucero Date: Wed, 8 Feb 2017 11:54:43 +0000 Message-ID: To: Ferruh Yigit Cc: dev Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2017 11:54:45 -0000 Hi Ferruh, On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit wrote: > Hi Alejandro, > > On 1/18/2017 12:27 PM, Alejandro Lucero wrote: > > For using a DPDK app when iommu is enabled, it requires to > > add iommu=pt to the kernel command line. But using igb_uio driver > > makes DMAR errors because the device has not an IOMMU domain. > > Please help to understand the scope of the problem, > > After reading your reply, I realize I could have explained it better. First of all, this is related to SRIOV, exactly when the VFs are created. > 1- How can you re-produce the problem? > > Using a VF from a Intel card by a DPDK app in the host and a kernel >= 3.15. Although usually VFs are assigned to VMs, it could also be an option to use VFs by the host. BTW, I did not try to reproduce the problem with an Intel card. I triggered this problem with an NFP, but because the problem behind, I bet that is going to happen for an Intel one as well. > 2- What happens get DMAR errors, is it prevents device work or some > annoying error messages? > A DMAR error implies the device can not access to the DMA address given by the host. I have experienced several situations where it is just that device not being able to work at all, but it also has more global implications and you need to reboot the system because it is unreliable. I think it depends on how these DMAR errors are handled, but in any case, this is a bad thing. > > 3- Can you please share the error messages? > With this problem you can expect something like this: 559.163874] DMAR: DRHD: handling fault status reg 2 [ 559.165427] DMAR: DMAR:[DMA Read] Request device [82:08.0] fault addr e7b73b000 [ 559.165427] DMAR:[fault reason 02] Present bit in context entry is clear [ 568.367417] DMAR: DRHD: handling fault status reg 102 [ 568.369025] DMAR: DMAR:[DMA Read] Request device [82:08.1] fault addr ebb73b000 [ 568.369025] DMAR:[fault reason 02] Present bit in context entry is clear [ 571.773944] DMAR: DRHD: handling fault status reg 202 [ 571.775550] DMAR: DMAR:[DMA Read] Request device [82:08.2] fault addr efb73b000 [ 571.775550] DMAR:[fault reason 02] Present bit in context entry is clear [ 575.039654] DMAR: DRHD: handling fault status reg 302 [ 575.041259] DMAR: DMAR:[DMA Read] Request device [82:08.3] fault addr f3b73b000 [ 575.041259] DMAR:[fault reason 02] Present bit in context entry is clear There are different DMAR errors, sometimes referring to a specific address being wrong. In this case it is related to the device not having a context or a IOMMU domain. Also note we got these errors for different devices/VFs. This was with a DPDK app using several VFs. > > > > > > Since kernel 3.15, iommu=pt requires to use the internal kernel > > DMA API for attaching the device to the IOMMU 1:1 mapping, aka > > si_domain. Previous versions did attach the device to that > > domain when intel iommu notifier was called. > > Again, what is not working since 3.15? > This specific case, yes. With older kernels, when VFs are created, IOMMU code is executed (notifier chain callback) and if iommu=pt, the VF is attached to the si_domain, this is the 1:1 mapping. But this has changed with newer kernels, and after VFs are created they have no IOMMU domain at all. The kernel expects the driver to implicitly create such a domain when the kernel DMA API is used. > > > > > This is not a problem if the driver does later some call to the > > DMA API because the mapping can be done then. But DPDK apps do > > not use that DMA API at all. > > Is this same/similar with: http://dpdk.org/dev/patchwork/patch/12654/ > > That case was another issue regarding IOMMU and iommu=pt. The problem there was when you detach a VF from a VM, but the VF was initially attached to the si_domain because the kernel did so. The patch helped to attach the VF again to that domain when binding to the UIO. Looking at that patch now (I did comment on it then), it just solved the problem if the VF was detach form the UIO, something that could be easily forgotten or simply not done because, apparently, it is not needed. What about to use VFIO? With that previous patch, it was not enough. I do not remember the details now, and I'm not sure if VFIO created another IOMMU domain if the device had one, but it could leave the device without an IOMMU domain after the first use. In this particular case, VFIO would work, because the device gets its own IOMMU domain. But there are two main problems if this is not fixed when using UIO: 1) UIO is one of the two options for working with IOMMU. We all agree VFIO is the right one for IOMMU, but as long as UIO is still an option, that should be fixed. 2) Some installations need to work with and without IOMMU. Having same module for both cases makes things simpler and therefore they use UIO instead of VFIO. > > > > Doing this dma map and unmap is harmless even when iommu is not > > enabled at all. > > > > Signed-off-by: Alejandro Lucero > <...> > > Thanks, > ferruh > > >