From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f46.google.com (mail-wm0-f46.google.com [74.125.82.46]) by dpdk.org (Postfix) with ESMTP id 5D54C2BE3 for ; Tue, 15 Mar 2016 15:40:17 +0100 (CET) Received: by mail-wm0-f46.google.com with SMTP id l124so13683755wmf.1 for ; Tue, 15 Mar 2016 07:40:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MURofneNFQHa1T1fCKl8TtSD5qYC9xfkxoP6PBDDvAA=; b=v7p9gd1VMOrGLIvAU2TZwcL37/xBYCdKq4GkWt0KVsjwVuUZxkH+UABIOF+q848IyD +nKTB9z3tUss8k5vtKTfhCrI6xY+joFxlMa7+64yTwF9rhGbDtGpe7lHpiAoVYtdxOWI KJe6StjmqRO/vx6FC3tLTHIG2ehPZkSIWue23IyovGs6cqSFSEuWVf4kD/4iYQIlipnj 1gTmNontVCrZF4VtjBIpmKEnsJKGKy+NlKTn/MNRQEr5nG4I+uDme4pv6167WPNC/3YR /J+CVv8uSjFraBSZZ6KUEyI1AufxBaS3nCVjVzTz859M6isrnjygTZgAaVI952vfEmTJ 893A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MURofneNFQHa1T1fCKl8TtSD5qYC9xfkxoP6PBDDvAA=; b=EZr6UCBAGe6Yvt0vPgC/P0HiVPt4TQeQfhbQaXjpRm4uBKoq2VcXpZg0xvkqe3smK3 dU7NBafTfKV7vlmFnvphThjTe2EZfOgYjNMjXIfIRh3lgZjpJadspuEPfnvgdSLgcjtj YLyrWinFb5syOn2oVdy4/r+ho32etU0UkToq9INi3x3CaMPgtlimxhHspdgGSEC1kkjS jTJfgSsqtUasnqW8damlUYTJA1OFsf9tB5chehF7pV+A+vw8vqhhpVMaUxH2U7SgdTxM 1AA0bOicw2qzE9MngaMcNuXdgmqKveNrsJqGmz8X3rGPp2xt/TkAuUsO1kISUQAQOXvc /3mw== X-Gm-Message-State: AD7BkJIlTk3E5J0tPHD4J4JhC553MOVihcDM0wH8+m7Y/Ha7Ncx7wdAVRdajr0MTeluzwA== X-Received: by 10.194.83.134 with SMTP id q6mr30389975wjy.131.1458052817230; Tue, 15 Mar 2016 07:40:17 -0700 (PDT) Received: from anpa-dpdk-2.lab.semihalf.com (cardhu.semihalf.com. [213.17.239.108]) by smtp.gmail.com with ESMTPSA id da6sm27361632wjb.24.2016.03.15.07.40.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 15 Mar 2016 07:40:16 -0700 (PDT) From: Jan Medala To: dev@dpdk.org Cc: matua@amazon.com, Jan Medala , Jakub Palider Date: Tue, 15 Mar 2016 15:40:07 +0100 Message-Id: <1458052810-18418-2-git-send-email-jan@semihalf.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1458052810-18418-1-git-send-email-jan@semihalf.com> References: <1458052810-18418-1-git-send-email-jan@semihalf.com> Subject: [dpdk-dev] [PATCH v7 1/4] ena: Amazon ENA documentation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Mar 2016 14:40:17 -0000 Signed-off-by: Alexander Matushevsky Signed-off-by: Jan Medala Signed-off-by: Jakub Palider --- MAINTAINERS | 8 ++ doc/guides/nics/ena.rst | 251 +++++++++++++++++++++++++++++++++++++++++++ doc/guides/nics/index.rst | 1 + doc/guides/nics/overview.rst | 116 ++++++++++---------- 4 files changed, 318 insertions(+), 58 deletions(-) create mode 100644 doc/guides/nics/ena.rst diff --git a/MAINTAINERS b/MAINTAINERS index f10b26a..d37aad6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -260,6 +260,14 @@ Linux AF_PACKET M: John W. Linville F: drivers/net/af_packet/ +Amazon ena +M: Jan Medala +M: Jakub Palider +M: Netanel Belgazal +M: Evgeny Schemeilin +F: drivers/net/ena/ +F: doc/guides/nics/ena.rst + Chelsio cxgbe M: Rahul Lakkireddy F: drivers/net/cxgbe/ diff --git a/doc/guides/nics/ena.rst b/doc/guides/nics/ena.rst new file mode 100644 index 0000000..9f93848 --- /dev/null +++ b/doc/guides/nics/ena.rst @@ -0,0 +1,251 @@ +.. BSD LICENSE + + Copyright (c) 2015-2016 Amazon.com, Inc. or its affiliates. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Amazon.com, Inc. nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +ENA Poll Mode Driver +==================== + +The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic +Network Adapter (ENA) family. + +Overview +-------- + +The ENA driver exposes a lightweight management interface with a +minimal set of memory mapped registers and an extendable command set +through an Admin Queue. + +The driver supports a wide range of ENA adapters, is link-speed +independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, +etc.), and it negotiates and supports an extendable feature set. + +ENA adapters allow high speed and low overhead Ethernet traffic +processing by providing a dedicated Tx/Rx queue pair per CPU core. + +The ENA driver supports industry standard TCP/IP offload features such +as checksum offload and TCP transmit segmentation offload (TSO). + +Receive-side scaling (RSS) is supported for multi-core scaling. + +Some of the ENA devices support a working mode called Low-latency +Queue (LLQ), which saves several more microseconds. + +Management Interface +-------------------- + +ENA management interface is exposed by means of: + +* Device Registers +* Admin Queue (AQ) and Admin Completion Queue (ACQ) + +ENA device memory-mapped PCIe space for registers (MMIO registers) +are accessed only during driver initialization and are not involved +in further normal device operation. + +AQ is used for submitting management commands, and the +results/responses are reported asynchronously through ACQ. + +ENA introduces a very small set of management commands with room for +vendor-specific extensions. Most of the management operations are +framed in a generic Get/Set feature command. + +The following admin queue commands are supported: + +* Create I/O submission queue +* Create I/O completion queue +* Destroy I/O submission queue +* Destroy I/O completion queue +* Get feature +* Set feature +* Get statistics + +Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature +properties. + +Data Path Interface +------------------- + +I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx +SQ correspondingly). Each SQ has a completion queue (CQ) associated +with it. + +The SQs and CQs are implemented as descriptor rings in contiguous +physical memory. + +Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor + +The driver supports multi-queue for both Tx and Rx. + +Configuration information +------------------------- + +**DPDK Configuration Parameters** + + The following configuration options are available for the ENA PMD: + + * **CONFIG_RTE_LIBRTE_ENA_PMD** (default y): Enables or disables inclusion + of the ENA PMD driver in the DPDK compilation. + + + * **CONFIG_RTE_LIBRTE_ENA_DEBUG_INIT** (default y): Enables or disables debug + logging of device initialization within the ENA PMD driver. + + * **CONFIG_RTE_LIBRTE_ENA_DEBUG_RX** (default n): Enables or disables debug + logging of RX logic within the ENA PMD driver. + + * **CONFIG_RTE_LIBRTE_ENA_DEBUG_TX** (default n): Enables or disables debug + logging of TX logic within the ENA PMD driver. + + * **CONFIG_RTE_LIBRTE_ENA_COM_DEBUG** (default n): Enables or disables debug + logging of low level tx/rx logic in ena_com(base) within the ENA PMD driver. + +**ENA Configuration Parameters** + + * **Number of Queues** + + This is the requested number of queues upon initialization, however, the actual + number of receive and transmit queues to be created will be the minimum between + the maximal number supported by the device and number of queues requested. + + * **Size of Queues** + + This is the requested size of receive/transmit queues, while the actual size + will be the minimum between the requested size and the maximal receive/transmit + supported by the device. + +Building DPDK +------------- + +See the :ref:`DPDK Getting Started Guide for Linux ` for +instructions on how to build DPDK. + +By default the ENA PMD library will be built into the DPDK library. + +For configuring and using UIO and VFIO frameworks, please also refer :ref:`the +documentation that comes with DPDK suite `. + +Supported ENA adapters +---------------------- + +Current ENA PMD supports the following ENA adapters including: + +* ``1d0f:ec20`` - ENA VF +* ``1d0f:ec21`` - ENA VF with LLQ support + +Supported Operating Systems +--------------------------- + +Any Linux distribution fulfilling the conditions described in ``System Requirements`` +section of :ref:`the DPDK documentation ` or refer to *DPDK Release Notes*. + +Supported features +------------------ + +* Jumbo frames up to 9K +* Port Hardware Statistics +* IPv4/TCP/UDP checksum offload +* TSO offload +* Multiple receive and transmit queues +* RSS +* Low Latency Queue for Tx + +Unsupported features +-------------------- + +The features supported by the device and not yet supported by this PMD include: + +* Asynchronous Event Notification Queue (AENQ) + +Prerequisites +------------- + +#. Prepare the system as recommended by DPDK suite. This includes environment + variables, hugepages configuration, tool-chains and configuration + +#. Insert igb_uio kernel module using the command 'modprobe igb_uio' + +#. Bind the intended ENA device to igb_uio module + + +At this point the system should be ready to run DPDK applications. Once the +application runs to completion, the ENA can be detached from igb_uio if necessary. + +Usage example +------------- + +This section demonstrates how to launch **testpmd** with Amazon ENA +devices managed by librte_pmd_ena. + +#. Load the kernel modules: + + .. code-block:: console + + modprobe uio + insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko + + .. note:: + + Currently Amazon ENA PMD driver depends on igb_uio user space I/O kernel module + +#. Mount and request huge pages: + + .. code-block:: console + + mount -t hugetlbfs nodev /mnt/hugepages + echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages + +#. Bind UIO driver to ENA device (using provided by DPDK binding tool): + + .. code-block:: console + + ./tools/dpdk_nic_bind.py --bind=igb_uio 0000:02:00.1 + +#. Start testpmd with basic parameters: + + .. code-block:: console + + ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i + + Example output: + + .. code-block:: console + + [...] + EAL: PCI device 0000:02:00.1 on NUMA socket -1 + EAL: probe driver: 1d0f:ec20 rte_ena_pmd + EAL: PCI memory mapped at 0x7f9b6c400000 + PMD: eth_ena_dev_init(): Initializing 0:2:0.1 + Interactive-mode selected + Configuring Port 0 (socket 0) + Port 0: 00:00:00:11:00:01 + Checking link statuses... + Port 0 Link Up - speed 10000 Mbps - full-duplex + Done + testpmd> diff --git a/doc/guides/nics/index.rst b/doc/guides/nics/index.rst index 0b353a8..d060265 100644 --- a/doc/guides/nics/index.rst +++ b/doc/guides/nics/index.rst @@ -39,6 +39,7 @@ Network Interface Controller Drivers bnx2x cxgbe e1000em + ena enic fm10k i40e diff --git a/doc/guides/nics/overview.rst b/doc/guides/nics/overview.rst index 2d4f014..28237e9 100644 --- a/doc/guides/nics/overview.rst +++ b/doc/guides/nics/overview.rst @@ -74,74 +74,74 @@ Most of these differences are summarized below. .. table:: Features availability in networking drivers - ==================== = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = - Feature a b b b c e e i i i i i i i i i i f f m m m n n p r s v v v x - f n n o x 1 n 4 4 4 4 g g x x x x m m l l p f u c i z i i m e - p x x n g 0 i 0 0 0 0 b b g g g g 1 1 x x i p l a n e r r x n - a 2 2 d b 0 c e e e e v b b b b 0 0 4 5 p l p g d t t n v - c x x i e 0 . v v f e e e e k k e a i i e i - k v n . f f . v v . t o o t r - e f g . . . f f . a . 3 t - t v v v v v 2 v - e e e e e e - c c c c c c - ==================== = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = - link status X X X X - link status event X X - Rx interrupt X X X X - queue start/stop X X X X X X - MTU update X - jumbo frame X X X X X - scattered Rx X X X X X X + ==================== = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = + Feature a b b b c e e e i i i i i i i i i i f f m m m n n p r s v v v x + f n n o x 1 n n 4 4 4 4 g g x x x x m m l l p f u c i z i i m e + p x x n g 0 a i 0 0 0 0 b b g g g g 1 1 x x i p l a n e r r x n + a 2 2 d b 0 c e e e e v b b b b 0 0 4 5 p l p g d t t n v + c x x i e 0 . v v f e e e e k k e a i i e i + k v n . f f . v v . t o o t r + e f g . . . f f . a . 3 t + t v v v v v 2 v + e e e e e e + c c c c c c + ==================== = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = + link status X X X X + link status event X X + Rx interrupt X X X X + queue start/stop X X X X X X X + MTU update X X + jumbo frame X X X X X X + scattered Rx X X X X X X X LRO - TSO X X X X X - promiscuous mode X X X X X X - allmulticast mode X X X X X X - unicast MAC filter X X X X - multicast MAC filter X X X X - RSS hash X X X X X - RSS key update X X X X - RSS reta update X X X X - VMDq X X - SR-IOV X X - DCB X X - VLAN filter X X X X - ethertype filter X X + TSO X X X X X X + promiscuous mode X X X X X X + allmulticast mode X X X X X X + unicast MAC filter X X X X + multicast MAC filter X X X X + RSS hash X X X X X X + RSS key update X X X X X + RSS reta update X X X X X + VMDq X X + SR-IOV X X X + DCB X X + VLAN filter X X X X + ethertype filter X X n-tuple filter SYN filter - tunnel filter X X + tunnel filter X X flexible filter - hash filter X X X X - flow director X X - flow control X X X + hash filter X X X X + flow director X X + flow control X X X rate limitation - traffic mirroring X X - CRC offload X X X - VLAN offload X X X - QinQ offload X X - L3 checksum offload X X X - L4 checksum offload X X X - inner L3 checksum X X - inner L4 checksum X X - packet type parsing X X X - timesync X X - basic stats X X X X X X - extended stats X X X X - stats per queue X X + traffic mirroring X X + CRC offload X X X X + VLAN offload X X X X + QinQ offload X X + L3 checksum offload X X X X + L4 checksum offload X X X X + inner L3 checksum X X X + inner L4 checksum X X X + packet type parsing X X X + timesync X X + basic stats X X X X X X X + extended stats X X X X X + stats per queue X X EEPROM dump registers dump - multiprocess aware X X X X - BSD nic_uio X X X X X - Linux UIO X X X X X - Linux VFIO X X X X X - other kdrv X + multiprocess aware X X X X + BSD nic_uio X X X X X + Linux UIO X X X X X X + Linux VFIO X X X X X + other kdrv X ARMv7 ARMv8 Power8 TILE-Gx - x86-32 X X X X X - x86-64 X X X X X X - usage doc X X + x86-32 X X X X X X + x86-64 X X X X X X X + usage doc X X design doc perf doc - ==================== = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = + ==================== = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = -- 1.9.1