From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 604C3A0597; Wed, 8 Apr 2020 10:34:39 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AB6C91C202; Wed, 8 Apr 2020 10:30:10 +0200 (CEST) Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) by dpdk.org (Postfix) with ESMTP id D32051C1E1 for ; Wed, 8 Apr 2020 10:30:03 +0200 (CEST) Received: by mail-lj1-f170.google.com with SMTP id q19so6646174ljp.9 for ; Wed, 08 Apr 2020 01:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gc7V4sTSOgeD1dMtWciT8dmyT5xWcLtVV7uSLTY6Pdc=; b=mCY9E4feEdJufFJV3b/S4orMD9JjFx87vVGiQVYgKEadShcT6EkZPF679AZtJD+9Y1 aVSzZ8JYcz3FcpIecYamDpc94o/ylGxr9oaNQlvG6rY00GBG3e8zsq2UH7x/XJV0E2U6 oIr3aJgK6AQiUbSOVbqM2epdqQ8jUjEZ63B6K3/twAhuWEuFhfCZlIJ9nYOAPNpsXyt+ VJ0zA97bh8GrxqrCBhDu5+9CR+uCbVCoXYSmrQlqB+Dg4g+nzhzbdzC7TCwOPtB9Rv9R 3QFGpiNY6mLexxFN5kv83OH3vGyXzPdSJBdCW+oNBt6O39qrhxhHcAR8WdYL0i0EzY1n JgDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gc7V4sTSOgeD1dMtWciT8dmyT5xWcLtVV7uSLTY6Pdc=; b=m19MlLM0te0dkesBc6R5t3Rzt7GsGClsWATJ2s6cLFeH0q1pRMqYA48mP5E/WAfAfy dRsf1kijkqXZvme95kg6URk9BgRA9b+BCRZ0JGsXoOHT2Bu+ACCzcr3TNbWGRewnZa6S v3YvDK0wj2oqwpPmGRSmH+LddEuyQrQtkvIKEiM5ZFX7hwNGiV7MFTr+gkHvtXqe+Vet OHxNX+W/Chq0Wtg6jVfc99TXuRQInkE35xqxQBK+82bgYGRHlMZfHhZb3JOzGQ+V8d3n esEJOcEswiG1ddp9IfIyVwpXpVHMg9g901oVX68vvq6fVSA6DA9HtXAX+vbdoPM5Zj+e tkVg== X-Gm-Message-State: AGi0PuYXqrGKNC3smsYn9TEQQkTPqInndXJTj0DT/5X24T9WBIy0gQbK fXBgPy1OCVsz6EXQ65Hc4z6TH+cqWho= X-Google-Smtp-Source: APiQypJz3uf1YwtBp2wuWW3Hv/7Kmk8fZFfEcZge43VMuX4uiZNcNamGk5ZcDi9nFlB4AAENK8pb0w== X-Received: by 2002:a2e:9cce:: with SMTP id g14mr4137717ljj.161.1586334603167; Wed, 08 Apr 2020 01:30:03 -0700 (PDT) Received: from mkPC.semihalf.local (193-106-246-138.noc.fibertech.net.pl. [193.106.246.138]) by smtp.gmail.com with ESMTPSA id e8sm765685lja.3.2020.04.08.01.30.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Apr 2020 01:30:02 -0700 (PDT) From: Michal Krawczyk To: dev@dpdk.org Cc: mw@semihalf.com, mba@semihalf.com, gtzalik@amazon.com, evgenys@amazon.com, igorch@amazon.com, ferruh.yigit@intel.com, arybchenko@solarflare.com, Michal Krawczyk Date: Wed, 8 Apr 2020 10:29:20 +0200 Message-Id: <20200408082921.31000-30-mk@semihalf.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200408082921.31000-1-mk@semihalf.com> References: <20200408082921.31000-1-mk@semihalf.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v3 29/30] doc: add notes on ENA usage on metal instances X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" As AWS metal instances are supporting IOMMU, the usage of igb_uio or vfio-pci can lead to a problems (when to use which module), especially that the vfio-pci isn't supporting SMMU on arm64. To clear up the problem of using those modules in various setup conditions (with or without IOMMU) on metal instances, more detailed explanation was added. Signed-off-by: Michal Krawczyk Reviewed-by: Igor Chauskin Reviewed-by: Guy Tzalik --- doc/guides/nics/ena.rst | 48 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/doc/guides/nics/ena.rst b/doc/guides/nics/ena.rst index 0b9622ac85..bec97c3326 100644 --- a/doc/guides/nics/ena.rst +++ b/doc/guides/nics/ena.rst @@ -187,11 +187,55 @@ Prerequisites echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode -#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module. + To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag + ``CONFIG_VFIO_NOIOMMU``. +#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module. At this point the system should be ready to run DPDK applications. Once the -application runs to completion, the ENA can be detached from igb_uio if necessary. +application runs to completion, the ENA can be detached from attached module if +necessary. + +**Note about usage on \*.metal instances** + +On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64 +hosts. + +* x86_64 (e.g. c5.metal, i3.metal): + IOMMU should be disabled by default. In that situation, the ``igb_uio`` can + be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please + see above). + + When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this + feature, while ``vfio-pci`` should work without any changes. + To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file + ``/etc/default/grub`` with the below extra boot arguments:: + + iommu=1 intel_iommu=on + + Then, make the changes live by executing as a root:: + + # grub2-mkconfig > /boot/grub2/grub.cfg + + Finally, reboot should result in IOMMU being enabled. + +* arm64 (a1.metal): + IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't + supporting SMMU, which is implementation of IOMMU for arm64 architecture and + ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those + hosts, one must disable IOMMU. This can be done by updating + ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot + argument:: + + iommu.passthrough=1 + + Then, make the changes live by executing as a root:: + + # grub2-mkconfig > /boot/grub2/grub.cfg + + Finally, reboot should result in IOMMU being disabled. + Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be + working in no-IOMMU mode (please see above). Usage example ------------- -- 2.20.1