DPDK patches and discussions
 help / color / mirror / Atom feed
From: Michal Krawczyk <mk@semihalf.com>
To: dev@dpdk.org
Cc: mw@semihalf.com, mba@semihalf.com, gtzalik@amazon.com,
	evgenys@amazon.com, igorch@amazon.com,
	Michal Krawczyk <mk@semihalf.com>
Subject: [dpdk-dev] [PATCH 27/29] doc: add notes on ENA usage on metal instances
Date: Fri, 27 Mar 2020 11:18:21 +0100	[thread overview]
Message-ID: <20200327101823.12646-28-mk@semihalf.com> (raw)
In-Reply-To: <20200327101823.12646-1-mk@semihalf.com>

As AWS metal instances are supporting IOMMU, the usage of igb_uio or
vfio-pci can lead to a problems (when to use which module), especially
that the vfio-pci isn't supporting SMMU on arm64.

To clear up the problem of using those modules in various setup
conditions (with or without IOMMU) on metal instances, more detailed
explanation was added.

Signed-off-by: Michal Krawczyk <mk@semihalf.com>
Reviewed-by: Igor Chauskin <igorch@amazon.com>
Reviewed-by: Guy Tzalik <gtzalik@amazon.com>
---
 doc/guides/nics/ena.rst | 48 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 46 insertions(+), 2 deletions(-)

diff --git a/doc/guides/nics/ena.rst b/doc/guides/nics/ena.rst
index a953068cac..62e05e62e6 100644
--- a/doc/guides/nics/ena.rst
+++ b/doc/guides/nics/ena.rst
@@ -184,11 +184,55 @@ Prerequisites
 
      echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 
-#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
+   To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag
+   ``CONFIG_VFIO_NOIOMMU``.
 
+#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
 
 At this point the system should be ready to run DPDK applications. Once the
-application runs to completion, the ENA can be detached from igb_uio if necessary.
+application runs to completion, the ENA can be detached from attached module if
+necessary.
+
+**Note about usage on \*.metal instances**
+
+On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64
+hosts.
+
+* x86_64 (e.g. c5.metal, i3.metal):
+   IOMMU should be disabled by default. In that situation, the ``igb_uio`` can
+   be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please
+   see above).
+
+   When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this
+   feature, while ``vfio-pci`` should work without any changes.
+   To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file
+   ``/etc/default/grub`` with the below extra boot arguments::
+
+    iommu=1 intel_iommu=on
+
+   Then, make the changes live by executing as a root::
+
+    # grub2-mkconfig > /boot/grub2/grub.cfg
+
+   Finally, reboot should result in IOMMU being enabled.
+
+* arm64 (a1.metal):
+   IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't
+   supporting SMMU, which is implementation of IOMMU for arm64 architecture and
+   ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those
+   hosts, one must disable IOMMU. This can be done by updating
+   ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot
+   argument::
+
+    iommu.passthrough=1
+
+   Then, make the changes live by executing as a root::
+
+    # grub2-mkconfig > /boot/grub2/grub.cfg
+
+   Finally, reboot should result in IOMMU being disabled.
+   Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be
+   working in no-IOMMU mode (please see above).
 
 Usage example
 -------------
-- 
2.20.1


  parent reply	other threads:[~2020-03-27 10:33 UTC|newest]

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-27 10:17 [dpdk-dev] [PATCH 00/29] Update ENA driver to v2.1.0 Michal Krawczyk
2020-03-27 10:17 ` [dpdk-dev] [PATCH 01/29] net/ena: check if size of buffer is at least 1400B Michal Krawczyk
2020-03-27 10:55   ` Andrew Rybchenko
2020-03-31  9:47     ` Michał Krawczyk
2020-03-27 14:51   ` Stephen Hemminger
2020-03-31  9:48     ` Michał Krawczyk
2020-03-27 10:17 ` [dpdk-dev] [PATCH 02/29] net/ena/base: make allocation macros thread-safe Michal Krawczyk
2020-03-27 14:54   ` Stephen Hemminger
2020-03-31  9:47     ` Michał Krawczyk
2020-03-27 10:17 ` [dpdk-dev] [PATCH 03/29] net/ena/base: prevent allocation of 0-sized memory Michal Krawczyk
2020-03-27 10:17 ` [dpdk-dev] [PATCH 04/29] net/ena/base: set default hash key Michal Krawczyk
2020-03-27 11:12   ` Andrew Rybchenko
2020-03-31  9:40     ` Michał Krawczyk
2020-03-31  9:51       ` Michał Krawczyk
2020-03-27 10:17 ` [dpdk-dev] [PATCH 05/29] net/ena/base: rework interrupt moderation Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 06/29] net/ena/base: remove extra properties strings Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 07/29] net/ena/base: add accelerated LLQ mode Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 08/29] net/ena/base: fix documentation of the functions Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 09/29] net/ena/base: fix indentation in cq polling Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 10/29] net/ena/base: add error logs when preparing Tx Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 11/29] net/ena/base: use 48-bit memory addresses in ena_com Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 12/29] net/ena/base: fix types for printing timestamps Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 13/29] net/ena/base: fix indentation of multiple defines Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 14/29] net/ena/base: update gen date and commit Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 15/29] net/ena: set IO ring size to the valid value Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 16/29] net/ena: refactor getting IO queues capabilities Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 17/29] net/ena: add support for large LLQ headers Michal Krawczyk
2020-03-27 11:20   ` Andrew Rybchenko
2020-03-31  9:42     ` Michał Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 18/29] net/ena: remove memory barriers before doorbells Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 19/29] net/ena: add Tx drops statistic Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 20/29] net/ena: disable meta caching Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 21/29] net/ena: refactor Rx path Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 22/29] net/ena: rework getting number of available descs Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 23/29] net/ena: limit refill threshold by fixed value Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 24/29] net/ena: use macros for ring idx operations Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 25/29] net/ena: refactor Tx path Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 26/29] net/ena: reuse 0 length Rx descriptor Michal Krawczyk
2020-03-27 11:29   ` Andrew Rybchenko
2020-03-31  9:45     ` Michał Krawczyk
2020-03-27 10:18 ` Michal Krawczyk [this message]
2020-03-27 10:18 ` [dpdk-dev] [PATCH 28/29] net/ena: update copyright date Michal Krawczyk
2020-03-27 10:18 ` [dpdk-dev] [PATCH 29/29] net/ena: update version of the driver to v2.1.0 Michal Krawczyk

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200327101823.12646-28-mk@semihalf.com \
    --to=mk@semihalf.com \
    --cc=dev@dpdk.org \
    --cc=evgenys@amazon.com \
    --cc=gtzalik@amazon.com \
    --cc=igorch@amazon.com \
    --cc=mba@semihalf.com \
    --cc=mw@semihalf.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).