From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0051.outbound.protection.outlook.com [104.47.42.51]) by dpdk.org (Postfix) with ESMTP id 730236CBB for ; Fri, 30 Sep 2016 14:07:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=cT4+xOg7bK23ifkDjV+Uc5RPVLSjaR+5OuBMsQ9pmho=; b=Sh8m3KWNoWawPzXeHvp6xO4nTMp79PhSKp/0HTWMYp3eUoFmXEk9FlgEah4VjamrfHuz0PxD5voYQ6QBAuiGeTwVLrFA//h8Rrio+CBu4IsGNl1+IfqcIZp17yz/cnqnPQyCtCQk6A7ImfON6KHDLqjQloP9Qa2gngIpv9iNizM= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Kamil.Rytarowski@cavium.com; Received: from cavium1.semihalf.local (31.172.191.173) by CY4PR07MB3063.namprd07.prod.outlook.com (10.172.116.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.549.15; Fri, 30 Sep 2016 12:07:05 +0000 From: Kamil Rytarowski To: CC: , , , , , , , Kamil Rytarowski Date: Fri, 30 Sep 2016 14:05:52 +0200 Message-ID: <1475237154-25388-14-git-send-email-krytarowski@caviumnetworks.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1475237154-25388-1-git-send-email-krytarowski@caviumnetworks.com> References: <1472230448-17490-1-git-send-email-krytarowski@caviumnetworks.com> <1475237154-25388-1-git-send-email-krytarowski@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [31.172.191.173] X-ClientProxiedBy: VI1PR07CA0001.eurprd07.prod.outlook.com (10.163.160.139) To CY4PR07MB3063.namprd07.prod.outlook.com (10.172.116.144) X-MS-Office365-Filtering-Correlation-Id: 50d0c9a1-4e4b-4c31-c88b-08d3e92a4c72 X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3063; 2:VaCXzL/kJDIfgkGhrSqZeErBmkJzH3fCcaVdg0CfvOwSTj9eb/X9OQC2SpGocuppwQYky7cWUaa7pfgea07bC8/4LL4qVO8yfOXsrAv6HQ4glReFH2Nt1bhdl7HUZRwadXp9pD2QnclJYg39bvnEOaY+3AkFvARl4jx9Ruv8ytxvG6Fmve5Wz8jqzkpiZFbS; 3:bHEGT622QVGaqSfvyEtP8UiSksH20DvAdoGUu6UWxzTSrbWI+crUxsvcanSprlktDuIqkuFt6iWP4CmJNpHNlqQRuuKB7We2286iQboiOdOu9MwBJ+540fYiVb/LGLW9; 25:GNMV0rvZhKD9babqZRRhJxbCm9m1oPTkb+tIciHSM1zMSrRbth/9Ld46O2heihQeE1h5B/NFQGJUG25r9WewLIbR40L4U2Tn6vdLPHS2Y/2IGu4hqLuNFubAyQGKrru+rQiCvHTpOhJOD89/kLfwx7Ui90NGEb3Wu7llUzjKlZ6K98V3EUfqDg1pT2ddQDzK/Ub037dMDmq1ntXqdVeJsjlnMb5Vy2kQb95BsyI+fKEOvcDLl87UcH5EecskgEK4dRjHv/b8e4Phu9ZtPa/XdrD/BFac6TYF6WsGFwKDKZyRTWEmN+zvc+PCJ4LovDKoGxjIZ/OtdgikFJxH/q/T63uOaB9ktrOqBATE3oI3PxNo1mZy+FW2ZDznZWo3yiysjyz+JYt8JToWxdYGEBtmf1ZjqURrpIuTwRDsCERKvbw= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY4PR07MB3063; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3063; 31:6yTR9KtQJq/BLxAldW4W+j5eDR0M5GhzVwLTluqnlm//mObt3WJxAWfLnfqbwgYJv4Ltcbz8rd83UVll8wZPKDHY2eUm+LJoCsEBcQfTmqL0Ixv4R6E6+QIvXm3Wd5CpznNr5LL4RgQco4vb+5ri095BftbNjXuTKUKsGniiuW+xyCPd7NKbDrmqoYbN0sMuJLggkh2bKCZlSEWeJvc+1YYYu8Qzft9vh3crB9qsJ50=; 20:fiRBbGdl86S2E9kfmJBN4BH3QiI/yyfZTspMKh9YUgpm4od+71WYrFrNz+TrlocXK8oej9l/tNwbbZl1CdF136B5kxTSBPGCdht2XZljkkuX/PrbiUX53dGsrAm7Br5iRvH3RYXLH+EF5+OueQZNtplVg6R6XbcvgyuPW9kX4q0NkZGIOxaNujWbyc6L96reo3F3fOwS6Sop5MgV+rmBqRfS6dHZfQaIPQvihUu7tPBBZaQGBF6Je4afwu8gqujVEgz+b5KOloBtf7MF6BSx5S/qiafmD2xZ0VvPrmK7IuMKgGMQ1ox/hKGD/JIj22xAGqFqc8E6KkHeP5L2gG33TAHUM061enMVp2c2UZxtGYosaezIlVYb5O82F2mXiLYNFbRg5wFNEUHYPBGs+hNK4Bp206+/baFjQWrGiIYSn1P2/qUb+6mcdS/rOgbYlTq7SS7hf1KCCTUZcsh5bSI3qTlQushKtk5Vsqv4+H1apY0BR5+yvVP/GjqXATld9pAnup/LR2n7T49plqlKbkmjdoK6IuJNHn4SFoBzYU1VgOukHWiYkLM5AVtlqLPg2JrnAPQrTQXt1pkEmHOH7QMvCV7D4Hy7QFYXAVob0BazNnM= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001); SRVR:CY4PR07MB3063; BCL:0; PCL:0; RULEID:; SRVR:CY4PR07MB3063; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3063; 4:k4Ghdtb5EkcxiTw00rr6Tsggop1tp46HduzQSkvk/K9yWCuxobXJUeY6vISFIYV4Wqzhf3buvPHS9GVSmnAAn7VHJdQ2dKmuKwGsZrtV3MsnU2Ncm9A8jwx2OkTeg/3xfJy3LG6GMB6RPljdBnhNvQWB0Zw6OrO2IuJjIjT0ZtI1Ki2FpHZFjosZbJSUT2Zj/iFvFi3psuiclJtjLd1MybaOhWio/owZhJjdDePIzErWPaYDm6eS44w8tB46rMaZneohYQmHneudDmppbz8HmjEz2ZfxDfzkWJw6KIMRLFc1fxd2seakovmb0d1tALAXCE4sD9pTDz9Z32mgqVaKxir7Nq4Dt89HaKpBHBkkt+UCcbk2zUdiRnMZ22j2iy8LujdZqC0m447h05WvKCx6AQ== X-Forefront-PRVS: 008184426E X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(189002)(199003)(92566002)(3846002)(66066001)(189998001)(47776003)(42882006)(36756003)(105586002)(19580395003)(97736004)(6916009)(6666003)(68736007)(42186005)(19580405001)(50986999)(50226002)(77096005)(76176999)(5003940100001)(6116002)(7736002)(2950100002)(101416001)(586003)(106356001)(7846002)(5660300001)(4001430100002)(305945005)(33646002)(229853001)(110136003)(8676002)(4326007)(81166006)(5890100001)(107886002)(81156014)(50466002)(2906002)(48376002)(2351001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3063; H:cavium1.semihalf.local; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR07MB3063; 23:ANf+a2szfjJ7VUOYX1idLRHlcPtThEEkMncsb4sip?= =?us-ascii?Q?wEKCzaCRTMh2NQ71ZxbNOE81piJqLS/HZBMRb310Vqk7AEMSwd2T20Jii8cC?= =?us-ascii?Q?TN2hq8PiZftXyGIS23VXlP3+0S+IwqcXGtoG4by8tMkb8K9Y08+hLbGaxcL6?= =?us-ascii?Q?FsSp9Lq/ZA+qKE8nfwA23QkmRIgtmW/3ALyz2zF3RbngPNRfgG7KM42HLD5d?= =?us-ascii?Q?sqdJmkOywc+gk5XEJT0i1Mn8NMJ4GW2sZit4qTKw+Qc84TdVWQ651AFsdwEj?= =?us-ascii?Q?7+j3JE7NUvLe5wTYL6HsgXdYVQGMotVHOP2DQsZc5hidLW8iFrZI3OJjabN6?= =?us-ascii?Q?ulpdSEncrjPPCpFkR7RIRl0w5t1kufyDl6GDG7rqsKZRJ/uTFJpMtpjBAJ2h?= =?us-ascii?Q?r3/ED7Q24X7O9+13xa5AqE+y+480EeI6OjdGu/4rn4VWPQPBS1LfqgIO1xwp?= =?us-ascii?Q?pAlViGfvyfSubIRJz97HU+9uxLNR4ZSAl1MJcfboB+5vVBKO3k18A4vLR3QV?= =?us-ascii?Q?mVI2NvvbQCtJdM/tslfajh6O+E/G6jb153RnaOwNLg0ZO4TINKKb+q0KAtY/?= =?us-ascii?Q?dRhNii/HS3RWEoUF9M0+IGDqjawZ/Va1FkTWgvp6KKX8zf5ZQRwSfyskVGy9?= =?us-ascii?Q?MU9O+JZ/9MKkK38LxuWvLRPFoKqjIqey4N6ozsdKwcYn7ejNXT2yzu46KLVf?= =?us-ascii?Q?LlTLg+Jeq9osbZgIehasE61jvT7y4BiPkcU7n8DS6BuRbCmHCXWcshjKvB4H?= =?us-ascii?Q?iGCUqqRJkmocY/qFulPaOBGX29tmgGgsZ62ni67p9QUmfF/ImhFSSCcVcad0?= =?us-ascii?Q?XnBYymbn1kmzMtTkvUsqVV0fTFWtFheM8YXwrXhnJAmTfZIXf4tTlZZTrYav?= =?us-ascii?Q?qI2cfy1DLXzdl+riliRt80LSIFF6mxUlk1MuKwoMBsKTRFUP+IyiLs7cR/ca?= =?us-ascii?Q?IRjmPvTWr9FqJeUuCbrszM5Mjf9IABZpa0S1rteU8BSsUim7Y3d3dLkWsgD7?= =?us-ascii?Q?8rwLKvxgQ1uZjbDKUbZ/HQ+6y3W/JV05VeA4/ZxB8y4jhVPjs/RYc1/HVs0h?= =?us-ascii?Q?aZY9nTMPDlym2sZQPKM/rw6AvuGS+XIpl8xr5sXNRbPmiZ0kj2Dd2K6GMVlo?= =?us-ascii?Q?TxZCMNkTOgJ3uOAm6k20yrVK2+Dx0unyXZqTTprJjEOWC3I4JlGsHkmVq8Fy?= =?us-ascii?Q?yy9lRV1t/5ZhbTVLzDoUSs58UnRcHv6+Y6M?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3063; 6:s/5dBHR9cDMKg1/XrXFoONEqYRFnUI+ea7ZFlgPOw1HdX9eZWu8lfrf3r3R9L9RvpiDMY+PjmUxU0zuht+PlKhqmKb3p0sx2NKhPhKtM0TTM5pHFLZJ5iJHdZTfpSsL9EYB7XXZ46KetE1CseTgyo2EgVgTHFSatMdRps0F622OrgpQzphtDwyg5ZX3zG7jXiTGvas4BNWsA1ALMJ0POfctOXbToLxSqFtHSofrqevgrGCXThqPWb77y4gVNzyQP2K3BF7YFBRkIt1DuQ5ePn/PHytCg19so0lRArW0RB5Q=; 5:Al+4KHvZXKSNTk0YGgFPQtVefQU5StXe8S3XjcYDNt4NQ7THPgst2+PJtgyxlJvKoD5MKkpps1bYK0yclrk0gT5Cy+zuXlpxgUuqTeJMOCy/09okgt8MEns46xkTGQzqwfwNtJQTs2x2fyRoru82TQ==; 24:2xPNIwTplEQOGdAgAAqa8+huYAFtSoTbxfa8ZsweuxbyCMzDiH2OZgTw6Mb3IuHu3ykPkHObQWT2PUELAoxVcTJBGq60XVVvEzEaIO2t+nE=; 7:2/2JOoOQH23mP4/NNuG1zMYWNGr3PD3IpZxNp3V+QrbSmfrw64pHpnRi7dpYUv/vHEWeoTOjIQHwx853+ExwHzq92g+96G/BAI/SFP6qdoYT24YnJA1kyew6AcLZDAJWz3WvHpcPlpMLWPWsc4EjAquHDc6kZzjUL5yDCBdxjJemyaCRYQnIFYY4eWnRVidgZeSSulzKsmydGxXrI3Z7MXvmXQamONUmNQEwIm53t9gBAdPxbX1NGJOxVEB0k3EpU7Z5Vk6IhM+Qe+MVKa7nZQhm7Pt78aVEKTNjSfBRWkzGSRDR0u64koedkduG12Wl SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2016 12:07:05.2997 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3063 Subject: [dpdk-dev] [PATCH v2 13/15] net/thunderx: document secondary queue set support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Sep 2016 12:07:09 -0000 From: Kamil Rytarowski Signed-off-by: Maciej Czekaj Signed-off-by: Kamil Rytarowski Signed-off-by: Zyta Szpak Signed-off-by: Slawomir Rosek Signed-off-by: Radoslaw Biernacki Signed-off-by: Jerin Jacob --- doc/guides/nics/thunderx.rst | 148 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 124 insertions(+), 24 deletions(-) diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst index acc05a4..9763bb6 100644 --- a/doc/guides/nics/thunderx.rst +++ b/doc/guides/nics/thunderx.rst @@ -56,6 +56,7 @@ Features of the ThunderX PMD are: - VLAN stripping - SR-IOV VF - NUMA support +- Multi queue set support (up to 96 queues (12 queue sets)) per port Supported ThunderX SoCs ----------------------- @@ -207,13 +208,13 @@ This section provides instructions to configure SR-IOV with Linux OS. .. code-block:: console 0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV) - ... - Kernel driver in use: thunder-nic - ... + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV) + ... + Kernel driver in use: thunder-nic + ... .. note:: @@ -230,18 +231,18 @@ This section provides instructions to configure SR-IOV with Linux OS. .. code-block:: console 0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Kernel driver in use: thunder-nicvf - ... + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Kernel driver in use: thunder-nicvf + ... 0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Kernel driver in use: thunder-nicvf - ... + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Kernel driver in use: thunder-nicvf + ... .. note:: @@ -322,6 +323,112 @@ This section provides instructions to configure SR-IOV with Linux OS. #. Refer to section :ref:`Running testpmd ` for instruction how to launch ``testpmd`` application. +Multiple Queue Set per DPDK port configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are two types of VFs: + +- Primary VF +- Secondary VF + +Each port consists of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port. +When a given port is configured to use more than 8 queues, it requires one (or more) secondary VF. +Each secondary VF adds 8 additional queues to the queue set. + +During PMD driver initialization, the primary VF's are enumerated by checking the +specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set). +They are at the beginning of VF list (the remain ones are secondary VF's). + +The primary VFs are used as master queue sets. Secondary VFs provide +additional queue sets for primary ones. If a port is configured for more then +8 queues than it will request for additional queues from secondary VFs. + +Secondary VFs cannot be shared between primary VFs. + +Primary VFs are present on the beginning of the 'Network devices using kernel +driver' list, secondary VFs are on the remaining on the remaining part of the list. + + .. note:: + + The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`. + We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility. + + .. note:: + + Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to + this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs. + + +Example device binding +~~~~~~~~~~~~~~~~~~~~~~ + +If a system has three interfaces, a total of 18 VF devices will be created +on a non-NUMA machine. + + .. note:: + + NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port. + + .. code-block:: console + + # tools/dpdk-devbind.py --status + + Network devices using DPDK-compatible driver + ============================================ + + + Network devices using kernel driver + =================================== + 0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + 0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + 0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic + 0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + + Other network devices + ===================== + 0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic + + +We want to bind two physical interfaces with 24 queues each device, we attach two primary VFs +and four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3). +We will choose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2). + + +#. Bind two primary VFs to the ``vfio-pci`` driver: + + .. code-block:: console + + tools/dpdk-devbind.py -b vfio-pci 0002:01:00.2 + tools/dpdk-devbind.py -b vfio-pci 0002:01:00.3 + +#. Bind four primary VFs to the ``vfio-pci`` driver: + + .. code-block:: console + + tools/dpdk-devbind.py -b vfio-pci 0002:01:01.7 + tools/dpdk-devbind.py -b vfio-pci 0002:01:02.0 + tools/dpdk-devbind.py -b vfio-pci 0002:01:02.1 + tools/dpdk-devbind.py -b vfio-pci 0002:01:02.2 + +The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage. + Limitations ----------- @@ -346,10 +453,3 @@ Maximum packet segments The ThunderX SoC family NICs support up to 12 segments per packet when working in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the frame size does not fit in the maximum number of segments. - -Limited VFs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ThunderX SoC family NICs has 128VFs and each VF has 8/8 queues -for RX/TX respectively. Current driver implementation has one to one mapping -between physical port and VF hence only limited VFs can be used. -- 1.9.1