From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0084.outbound.protection.outlook.com [104.47.40.84]) by dpdk.org (Postfix) with ESMTP id 8A1456A16 for ; Fri, 26 Aug 2016 18:55:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=LI19wyzsmJf9LOAnC3NeU27Whs8LIiUozWejHEMJJOU=; b=G6cpkAClwU9gUY3uooXGjUSm+yZsbjiUhKDvCEbbx1eakwE4JI1VJDStcrwLsJ3ljTser6r2dlPegjvmNL4ozuv/p0fT3Vush2ABAnREwUMUEOxVG3BQrBVgqxWN0/CCr9EPueeRVih/YXh/mM/6Mp4M2MITGKqaJFPtIZzhOEg= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Kamil.Rytarowski@cavium.com; Received: from cavium1.semihalf.local (31.172.191.173) by CY4PR07MB3064.namprd07.prod.outlook.com (10.172.116.145) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.549.15; Fri, 26 Aug 2016 16:55:20 +0000 From: Kamil Rytarowski To: CC: , , , , , Kamil Rytarowski Date: Fri, 26 Aug 2016 18:54:08 +0200 Message-ID: <1472230448-17490-14-git-send-email-krytarowski@caviumnetworks.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1472230448-17490-1-git-send-email-krytarowski@caviumnetworks.com> References: <1472230448-17490-1-git-send-email-krytarowski@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [31.172.191.173] X-ClientProxiedBy: HE1PR0201CA0008.eurprd02.prod.outlook.com (10.165.229.18) To CY4PR07MB3064.namprd07.prod.outlook.com (10.172.116.145) X-MS-Office365-Filtering-Correlation-Id: 67873df3-b402-4f79-7879-08d3cdd1c492 X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 2:2gCL4tZPaCtGjkZ+i+ZAnIpTBI0c88OO9jrIClDfepw7pHyQ96fGmaqut3pMEcamCSPZXaM3ExngLn+itR7YNai85bqNAim0pQj23JyPFf3d8BHX3qwrYsKQ0zvFNDYe+cUGwpyQoSwqAMzlMYhAUnF0SD3RXOO3CDPDN7Ol7CeCiywnd5ZKJPdmb/l0r0tS; 3:OoA6S4BsSF2nXdgarfseSSBrNHHK+x16yTdg83/lNPle0B4YSDulSjyZlYfXAOPmh5kuXHkUCxkuOCoge8utwrtmKHiDfX2saUu1pWsu8vRORIb5qplf46zCQ99/ANjL; 25:QcXWAwo4HRMeRV2Ovqeyw6M/Vu45rmq2aRL30VuB53+kKsy1Pvxw3otzou7CpDZAK4vS6gci2mvYiomXm1WcR0gDM+8qpK2YWSP1EJrFlEsnIEt7foRG+Pmj6wiFYk0xlT3LHGf+6Hl2FLY/qiVJDMvupwT9AktO5ExvwCKO/fNBMXm3Vd+yQ8g23AUCCVfJVsHEEj8Q8xcOw5c39yofvOt/zt2TCOVSp1cgLT7WjScR9GM9WQT3m8x0sHgFkOigeOgPVk21oTFe8CaK2Diy8VqdO00Yjr4A0JT+CFdE4488mXEo86dRduyoz5G6xbN/eeW7vPCoOtZSUxbJId2azD+TesDFDZUK7vKvsSwG10bh9OJlDrQYlSFBnwCATiqV2obLZ/SPVLNk1YrtJGMiWSzbg5O+YK+6qL7ZIQiVKGo= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY4PR07MB3064; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 31:uSLHYlWoQFssvyRnluXSzDfFgS8ijUKt/fFNCUtnrGRjiJBQqXpqRCWlAuRVV+NCozGK9ZC8qpJOip7jUOwaYHU1wEa2foxN6DoZBIZg37Rq3OGvXEUJhSmeAMRuRhQmPzIPvgkvMECqQOM/FLdag64WFWFvVbeM96CHkbSliRfiTAk0VrHH9TzKb4Z6snSdAuudoEsZ1ZimYenHSg1BHNOSPFpGN8sD1p8kixf/I/s=; 20:UeyV6WhAm2/mPPlWidvEO9InKcxTZUlAr4Df2rkZyM/TIsEC+2DN7IFjva+sIdpitN3m/gYcZ+E0LO1R/MJS6nVvkB2ngePU6615aFE1ZJl74P9u0uHmeHu/RNmlKkW/Wt61EFBIv0PE5gZIQ7fGbuiGpej4MFDkogXlKLDGtk19oy3t76SdpTf/2d77EUPh9b0the+I/NNCHHyAsY2zI/o5UVcUZAMcI7z1O1JctFMjlZR3IbheosqxJ0Qjogk6gDQi3iJMw4q5l0kITreVHD/vuuxQ6D/iRUdlBxIcjStE7jlGZzQUrIvz5Qhym41HF9hpU1RGRl/BLrau/esvVNH92mGKx0arOhuKoPNeFn7+bYIfe+SpGWBkXs6L+QwfqksSyYFdQm8lR0oHk5CEquwVOGH3jyIcOKXwCDEHkfrnMRb6btV/YY09/297BS11kMZ3ODF4yiXpkAkZKP3DZUhlyOIOrvPWEco5TvNpNGmNUv/w7GpYhjRnAuvjT52mBt/F695wruTrUV0MsrFBnAxMnNJ15uD6LW4zfgA3AffuhCVtry1CrHuC1e1fI0GtSsIxo8ZpZBKSq5feGmpqTsBiYLj+K3a+e/UxZhi6QBg= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001); SRVR:CY4PR07MB3064; BCL:0; PCL:0; RULEID:; SRVR:CY4PR07MB3064; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 4:BH0NVFN1e9Rr3VG0/wC1BRqcg90tum7vdm8N7I+U+Rplt9phxyTjQD/8B+pLqLwQQQVOzczlHXlxSIh73S2gaTij7YwpYc5YrTl8XtpToxwJ1YjfGRKygIe8ctErWxp+D95voUv7Ux8KkPqCnCCYd/XrAWtEAMDkf8wDCMiOiTp5sxWOG+jzpg58ImovsEQUtNJXa0EhwvOBlQaNCdX/o2InEJF49RZxq4qxIV3gUZGp3L1JaRWWLiquZfrs85CNzDtm2lV+DInYWSHls/ot9pnwRno8oo3eNXvzv4vFXc5RVLfmExEKiwztj7Iwmb9WVrcSEW0oeluKtnE4MFVRUgodhGhjW8LCFsN9yoKe6UZM9Px7hIZ9KqzMfSpwnCPSbRZ4hbmBHYZxGty/CQbEzA== X-Forefront-PRVS: 00462943DE X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(189002)(199003)(81156014)(36756003)(77096005)(189998001)(2906002)(229853001)(2351001)(105586002)(8676002)(101416001)(92566002)(5660300001)(81166006)(6116002)(4001430100002)(2950100001)(7736002)(47776003)(4326007)(66066001)(586003)(50466002)(106356001)(50226002)(5003940100001)(19580405001)(19580395003)(305945005)(50986999)(110136002)(76176999)(7846002)(48376002)(68736007)(42186005)(5890100001)(3846002)(107886002)(33646002)(97736004); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3064; H:cavium1.semihalf.local; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR07MB3064; 23:A7W7oymqjaJhq9OrkY5DzJ/O02+7F2KPJ4wV3LSTN?= =?us-ascii?Q?JcZqJnlOlqcWyO0YuA37h1spvQTCFgg7j7V7dMXzmwnXy2faULrXXnwp8oNO?= =?us-ascii?Q?0fmstR1x9PmvWU0VdN2jwR4CwHqqoOI/M+MvzI81pZt6h7Q28VzzbST+Orum?= =?us-ascii?Q?CbD4HTn47/X3fs0pgtjcBxdGY0BEbR7735nCgow5FQomjVrV5o7H5TTKtMkJ?= =?us-ascii?Q?KXJYLSTObRmBBU8ap/qHeSHMEmSIHuuwwZNAf/N0TjpCF5SI+0Fjua+BjKFf?= =?us-ascii?Q?gsaa75xM1inzpAv+gGSa3ynih5XBh4+KQ2sWX0+lFW/QESWlh+dWxn72DDAK?= =?us-ascii?Q?xqcUbKCI0NSzey6w7guRtDbc6dW3gbAoGrnxhEARY9ChtlMnmQxjIG2oNbCs?= =?us-ascii?Q?sBZZzpeVGekLZlqaaQ6JDwAw59sfjLLragx0hADiP10Lb/PS8oBBrpgfQ+Ov?= =?us-ascii?Q?hCtSZXP/MR3rcYczEOpnsAsl/DhNiAKKMAQF0ZtW3p3pw0SEAt7ITLVSzvDw?= =?us-ascii?Q?UFu+OgT9wmN2fhWwv/ukmcZZPb+ZzM8kYCy+K8poNuVvRGgnNjDtbqHaN999?= =?us-ascii?Q?mjlkoJkp5R2XR9Pw+dGX40zVmve5nNLMaANFC1W3aeUKX+etGBRY4Wa1fXz2?= =?us-ascii?Q?SYQWoQeIbXNbTTfHdmFkwTYBumq6w3fluVuOTYdsbkkUacTPzcyafS7Jarmx?= =?us-ascii?Q?q5mw2UE2Qj/vV84lRaVpjbeP8M3iKTojO/s4am26CPXReHDWxJ6tQNG66Yhi?= =?us-ascii?Q?XSrthQhWhyu7Fz9shy4hfuE63lXiBvT7m6MddUiLjSFhTXN/O7Sx3RS+aoUL?= =?us-ascii?Q?BAKTM9gHKsQimGxLkOhBdL5MiPir1a8y1HXGC0pjJZ3k5pzOkfZoSqsiqQDj?= =?us-ascii?Q?pJLnhdvtsZv9bpvjhayxiTWu1hWbKJ0685nv8DzmcspcitcwpQFySpDNi9bT?= =?us-ascii?Q?JxWX+/lKA8S3aU43r7loxIF036uGy/k07Fj1FDuxmdAhBS0bZxK4UtWce/Q/?= =?us-ascii?Q?LWD/s2mVOdk2n4Iwe2n3OKI55vO4DbkYZyusxfML8bVsH5y2qHDXxI52KXp8?= =?us-ascii?Q?RTsdmdKc6O3NIEebYbeUmeDqyT+wzafGW/pUNRoT61u90cNLBV8ZkR0jr+n+?= =?us-ascii?Q?vytBXxLBvYR2bB9dOZ4WHZMVBRFQNjvHKwVQIDdB6/smFzOWuT74A=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3064; 6:zSP8tnNZndZhkoj0NZ65pQvE1aeKB+bEdIvStDyrE0hl4SLJpPBXx8tFhDPu2fEPlp+SJxVllropNJ18U6K6nEh3HgMKem0YsvRJ1QvWdO+/uaz/X2AWL9NxhxKDv2dZjE9wMmSwUugmrrHoHuppF1FvA23jvZXcwErzOhw3fdoFtq8JW6h7boK1WWZGI5jo5lU6oxjZ4d+YKLWv9sfW5DX89LRnTNtW7OGqq0egSiYtSpb/7WKDgMMTq0nJ19WeXFE57D5NicGSgWbnOue+MrknPokx3y+MYnlkuLnUNxk=; 5:vGsautakD69Evmgz0bzpGoRCdhjBNX6c8aRgKpFFCKUWkMTqaDEyscoQIwWZy87LVtV636vitwQWUyU2AZ8I7iEC+PemTq65Kp+4iOWxopiN+EHYz5ftj+NdH1/dIGbSL1gKwn6Rloe964cvKJP6sQ==; 24:ZZ6YHWZn4f1NvOWpUWve4X1QU88ewV6voZHknC1+eA5VN8SSSgkT9MXciUXm7vU8lzKGqQFib3+clambJzVI3pKgBNbbgUENooqaH/D8PEE=; 7:feyOnUglAJWhkVfSAIRQ+giHbBIVRDRm0ADV7zaTX/T0WGGzm85asmkidP+Zu4oL2YmCXPMdzaXyijVQkGbYsDtars1WzVu8BclQAP8broKixjjD8mfFddlw0a3CaTsj3zpYubgEgVXSym5Fc/4j3KstT3MJ2jLrTESnLy5DfPYH1fft437PmqXc6oYwgjCA2SyqEIAORcHqDz1tPP5F6SgzsWw8YvVQahhl02vQw5zfIxi5uhyq0hQHUCETs/5v SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2016 16:55:20.0498 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3064 Subject: [dpdk-dev] [PATCH 13/13] net/thunderx: document secondary queue set support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Aug 2016 16:55:27 -0000 From: Kamil Rytarowski Signed-off-by: Maciej Czekaj Signed-off-by: Kamil Rytarowski Signed-off-by: Zyta Szpak Signed-off-by: Slawomir Rosek Signed-off-by: Radoslaw Biernacki Signed-off-by: Jerin Jacob --- doc/guides/nics/thunderx.rst | 114 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 107 insertions(+), 7 deletions(-) diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst index acc05a4..2b6f1f0 100644 --- a/doc/guides/nics/thunderx.rst +++ b/doc/guides/nics/thunderx.rst @@ -56,6 +56,7 @@ Features of the ThunderX PMD are: - VLAN stripping - SR-IOV VF - NUMA support +- Multi queue set support (up to 96 queues (12 queue sets)) per port Supported ThunderX SoCs ----------------------- @@ -322,6 +323,112 @@ This section provides instructions to configure SR-IOV with Linux OS. #. Refer to section :ref:`Running testpmd ` for instruction how to launch ``testpmd`` application. +Multiple Queue Set per DPDK port configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are two types of VFs: + +- Primary VF +- Secondary VF + +Each port consist of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port. +In case port is configured to use more than 8 queues, then it requires one (or more) +secondary VF. Each secondary VF adds additional 8 queues to the queue set. + +During PMD driver initialization, the primary VF's are enumerated by checking the +specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set). +They are at the beginning of VF list (the remain ones are secondary VF's). + +The primary VFs are used as master queue sets. Secondary VFs provid +additional queue sets for primary ones. If a port is configured for more then +8 queues than it will request for additional queues from secondary VFs. + +Secondary VFs cannot be shared between primary VFs. + +Primary VFs are present on the beginning of the 'Network devices using kernel +driver' list, secondary VFs are on the remaining on the remaining part of the list. + + .. note:: + + The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`. + We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility. + + .. note:: + + Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to + this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs. + + +Example device binding +~~~~~~~~~~~~~~~~~~~~~~ + +If a system has three interfaces, a total of 18 VF devices will be created +on a non-NUMA machine. + + .. note:: + + NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port. + + .. code-block:: console + + # tools/dpdk-devbind.py --status + + Network devices using DPDK-compatible driver + ============================================ + + + Network devices using kernel driver + =================================== + 0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + 0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + 0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic + 0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + + Other network devices + ===================== + 0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic + + +We want to bind two physical interfaces with 24 queues each device, we attach two primary VFs +and four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3). +We will chose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2). + + +#. Bind two primary VFs to the ``vfio-pci`` driver: + + .. code-block:: console + + tools/dpdk-devbind.py -b vfio-pci 0002:01:00.2 + tools/dpdk-devbind.py -b vfio-pci 0002:01:00.3 + +#. Bind four primary VFs to the ``vfio-pci`` driver: + + .. code-block:: console + + tools/dpdk-devbind.py -b vfio-pci 0002:01:01.7 + tools/dpdk-devbind.py -b vfio-pci 0002:01:02.0 + tools/dpdk-devbind.py -b vfio-pci 0002:01:02.1 + tools/dpdk-devbind.py -b vfio-pci 0002:01:02.2 + +The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage. + Limitations ----------- @@ -346,10 +453,3 @@ Maximum packet segments The ThunderX SoC family NICs support up to 12 segments per packet when working in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the frame size does not fit in the maximum number of segments. - -Limited VFs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ThunderX SoC family NICs has 128VFs and each VF has 8/8 queues -for RX/TX respectively. Current driver implementation has one to one mapping -between physical port and VF hence only limited VFs can be used. -- 1.9.1