From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EC96A43B64; Wed, 21 Feb 2024 15:44:20 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 65CCC4064A; Wed, 21 Feb 2024 15:44:19 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by mails.dpdk.org (Postfix) with ESMTP id 38ADE402CE for ; Wed, 21 Feb 2024 15:44:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708526657; x=1740062657; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=H/3GSaikeQQJERlwSQ3SzT2sMaG1mEpqMaeFaCIMRLY=; b=j8+TJSKop4I0PRVhi8qx1HSXPskbYgaSwSfme30rUOZV1LmId8snldxF i69CsNC47kn+XrflJ0eNAmPuyIyTmfPVazUctBPLFWurCqxORZ0DFdW9/ 8++Mh8BL2kHNon7LyQqUWLNiAnpKfOs5g/xm6KgezANIyOsCWY9jelcn/ Qe9RsuxOVQGImA+TgRwyTS/xChDqepl28EbXLECHFlPCTksw06ADXOeCr eeVP3d1vStFzXJ0hSUqJLeu2Hkfl0RPe+iBdtSvVx49/sbdSOcYbamAfO SqQhQiGGntIhlMoKOsGEikXTVbMYp7KXpLWBwfUUuY51VvwHwMym+Zbfb Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10990"; a="3171257" X-IronPort-AV: E=Sophos;i="6.06,175,1705392000"; d="scan'208";a="3171257" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Feb 2024 06:44:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,175,1705392000"; d="scan'208";a="9750542" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by fmviesa004.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 21 Feb 2024 06:44:17 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Wed, 21 Feb 2024 06:44:16 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Wed, 21 Feb 2024 06:44:16 -0800 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.169) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 21 Feb 2024 06:44:15 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aLNZWYBgvRctgI3VfKGn8hw1YPCKBXTkdr4TDLQmU+mPDv5NPsqrrelXigkYHmt2KfqYD5c7NfqdCaSZORiH4yWqhL3d+bNCX1XbU1GsTFJ8pOewrT45lZOZjfSmLU/D0HA0nnlaj+Q76JGKkLAPlFSjE++qOlJeX2uzFSBJKkf4tanVPsEqVw1LLUy8uxgbcrGauuqmFwbB28gDxvHz8EfEH21F0kxt64YtzAaBaF8HZU/7H1Ey+KxDbu715QuPEjR0fhF2FCGVlII+VC4i/MOSpHrkH56bkqlKtQxp388NzDpDDn5CgrEIb75XuWf5e371B2FwhOXIblmgg3DsjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CdAfMF71HnD4B+SXX9XqtbxxroZwU6hIZVdi0mazmgk=; b=hiRPoLYsMWks8hJRjwwtnKeo7LPuZdo6R1gjopf9mW+wWInSv1hrLxwRjoYRM+TKxpg0KsgCZEEQ2GcaXxUyK5H9GPWxQeSHAgyQXrA7hf8qi6efoFc8UQ+qk/76zwgsc+7Z5+mH5YeZ5f+fl766BbgWEoThHK5hJvbSIBpIP5TYxYJQXa1yFL3LSyMDxFsgQJ3PMMs2DkAidYZqhcolQz24aeHBFDQrdkHlHywj5fsRkdSXzweZf4PJYeNdSa4KrOzmwSVQO0Ejj1UkIDy7ph8C+cUBqaCeRZZKbdHL4O/S6qjVjan2SC+yT0VEHkCWBKEbZYOaizQBIaHA2xdpYQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from PH7PR11MB8121.namprd11.prod.outlook.com (2603:10b6:510:234::14) by MW4PR11MB6811.namprd11.prod.outlook.com (2603:10b6:303:208::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.39; Wed, 21 Feb 2024 14:44:05 +0000 Received: from PH7PR11MB8121.namprd11.prod.outlook.com ([fe80::1a1:73a7:a80c:b694]) by PH7PR11MB8121.namprd11.prod.outlook.com ([fe80::1a1:73a7:a80c:b694%6]) with mapi id 15.20.7292.036; Wed, 21 Feb 2024 14:44:05 +0000 From: "Koikkara Reeny, Shibin" To: "Tahhan, Maryam" , "ferruh.yigit@amd.com" , "stephen@networkplumber.org" , "lihuisong@huawei.com" , "fengchengwen@huawei.com" , "liuyonglong@huawei.com" , "Marchand, David" , "Loftus, Ciara" CC: "dev@dpdk.org" , "Tahhan, Maryam" , "stable@dpdk.org" Subject: RE: [v9 1/3] docs: AF_XDP Device Plugin Thread-Topic: [v9 1/3] docs: AF_XDP Device Plugin Thread-Index: AQHaX2gGO/NLD5/wpk2P4WkK4gYUKbEU4DPw Date: Wed, 21 Feb 2024 14:44:05 +0000 Message-ID: References: <20240214170404.3318857-1-mtahhan@redhat.com> <20240214170404.3318857-2-mtahhan@redhat.com> In-Reply-To: <20240214170404.3318857-2-mtahhan@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH7PR11MB8121:EE_|MW4PR11MB6811:EE_ x-ms-office365-filtering-correlation-id: 294db4df-2eef-41b3-2882-08dc32eb8d85 x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: N5mpJXMKJg7Vc3J7SfKkYIcGfnG2MHkguVrC+IOa4FKig7bC6iUy1Ruc3j8PjBPjkXJrulfWvaKu8kJhWbJvrwm6k9QifAJaHrdcWbjlOVqakjdD/Yfp6HW2dAQmadyu6Dv271tI4+FDdVlwh3j8enLOimrX55BMwWS9N+XoR7w5w/kkfPfmuJzJsXB9yAQUCmiy+wqm/IaG+lbOQZmpMRQzCtssazUQQsqLUt/LrqawkKHjG9E7dv6Z1Geb9Kkoa5J+M+LmsRrPtSagTzK+6YpiKUSbUYjx5ZmfLW7IBmdaVk1G9QEU0DbGKu/KCAuae9Inn7aBxBr07+o0gBl2Bid40NPRe7IdkMlF2XJNKOlGvHNOxHeBI9rYLJvEqXraL+u78EeSGTM8HIiHmBLEeHY3wOiDFw0K1Fvwy5u/Kutmr1ldnMyWWlvbcUETqB8WCtymaWloa3W5oB26i2mNcRRJsx7/RggXA4w2THWaVOLBxhzMOpNT+NvMW1N6uNkWMBTYuKaVgEEFdmIsYEsOyeGdDJO+sWiB+Q9gwdkI+W+1bj01pXnrr81EgD4ZE+wke7XsPJXKj/19zsgaQFofM2hxb/mCZyPZ6rRvH1pUF/ZGZBjoxqraZlmtDawEnv43 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR11MB8121.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(921011)(38070700009); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?EI5ZjxyfFD1qtjQzMX7Uxf20ipdSr6tamiQhFpQ0Bxe7v/VIvSoTZ1nvk+SH?= =?us-ascii?Q?eHyGXIhqef8lcqFGZ3Ce06F0Yo53nniRHcf4nEXSXi/K+m8Wh3bWgEnOdPs7?= =?us-ascii?Q?aCrfGQVe85bNe5u1nGX86bC8oD8svmfbjVsKrvYrYIH7jmaxGraDdYL3hVCg?= =?us-ascii?Q?G7ss2nSfEtElX+FC7bs1NgO0iMuyWtCktcerMkf4Ozo9TIYwz/7/oAW25sxk?= =?us-ascii?Q?Do5QQdlOoTkTNQAF4HkwkA/9F0YRqORD3nzkSlbao9mnkQVjsgAH/b0YG+lT?= =?us-ascii?Q?G2+PMnU6Ya+fGHlmsnMeIjtmRNtnlYNxc5nqBGeXk5LSSftok04vrL/O6S8X?= =?us-ascii?Q?IgjWEtZBCbJh62w29xKxWXYmpgJw2V2YnU2SdhWxvclNAi3Xci1kL9macTZV?= =?us-ascii?Q?NMML1w7bHQ5CcaOXJjqXsTjPMWMrze5oJ34dPHPcJAQA/SPz7pOrHBdzWEEJ?= =?us-ascii?Q?m6NlAKkX0G4orAZ8Re89kbXF7DGlTOEMAOzfQioTOWLe3iUPgM5lPK2DJrrg?= =?us-ascii?Q?8ZtT6Q+a3fu/iGsyEEyeR6WPSCTS81K6FQD3xQdHvPYDDicBqVSTi6Oc/3kj?= =?us-ascii?Q?VlQC2kBb7cW8+N0+cHNQCGSxMU/+QzWm7zh6Y8X1kbT2JAMmBC+4JipWiZqs?= =?us-ascii?Q?GAJr9Hm7isBwFH5ZFrZkKzzYQT6n+2obsi12yjy8hjZcwy9guMlGa87vZW68?= =?us-ascii?Q?tMzaxf+Bb1bAwnY/AOZxXTDCtBl691grwzd+1uufinHGlXoZ82MUCxs3wG7s?= =?us-ascii?Q?MX0hTvIhUDVFf5Niv/fvUN97eY7vN8aUFHtk+xydwbaiFZCQANRuAByexDHT?= =?us-ascii?Q?nZC6ISechYM1RnuQTwrYG36QrbrQy8/a5p9pt+Utoik/tz6FpEefaj6ka4HB?= =?us-ascii?Q?jEtbf1AoyfvDNC5lWNgSAAa0kNcKOfiSstx5ihXOZlf7BOMQvta6LZ8RB0fH?= =?us-ascii?Q?HFdcbP1+uIoTthF7pzR5a7iPUDkgemLr+PyMwZ/QfekA8azcg1vFPK3jRG3y?= =?us-ascii?Q?KMCzxDSQpYuhRkptM/xTupfEl9Uk6I/3SKFapGTgwAnVplxw/0QkY8Qs7NpP?= =?us-ascii?Q?J/Gm6CRTBPjsK51S4jNyUxwD3nfpbKr9Pkq1rHpl9AdaVYxO+RuiNwX/SRQT?= =?us-ascii?Q?85OpQlxzq7vGATw3u+f3v7fG0EeJhK8Mf5zPzpnklvUZmmdE4NQ+/xdhXyDq?= =?us-ascii?Q?TDddYV0hcX+/Yl++ax6RkNubcupQRNQWNCcZE//U5LeuRo+BMnO8b4XtW778?= =?us-ascii?Q?VnrmsaDsvjuSI+2oh7g68qN/bkTqGmRtRZy7VvHVmxrdu+Irwy0VID+c8wp0?= =?us-ascii?Q?hxabz3uHm6W1DK3NC5QiRo4ozbQ8kXvPlvbLtBWevSv7MIuXr6VP2PrpUUme?= =?us-ascii?Q?/T4qmFrZJFeUSuWhRYI6zpQ34GiW3BAxuOss3N+t2O05JVaxAIxOFbLids5d?= =?us-ascii?Q?gYLaAwRxscFTs9LHgRJ/y74wVa40k/Yp7YgtUWEa8YTigvRJzVSGTwTW6zq3?= =?us-ascii?Q?ns8LS15lWC4geZnLoREgzhRkFGFo71ZAaBgwx6PVYEeMFuiO1C3nLyUgZ0Mm?= =?us-ascii?Q?R6xYMFdJyhxLjEKR0HYLuD+saCi/10GeEV0IKb5dLeGhsXjDgYrlvGrOefQs?= =?us-ascii?Q?CQ=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB8121.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 294db4df-2eef-41b3-2882-08dc32eb8d85 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Feb 2024 14:44:05.2335 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: OYTbfIYlJkmm3ML04ALZld/3riO5gCiMKbmzJu6K02U8FIKgKBfHfxuqq1O41W3khDYXJDXqAzod909NTpq39yAuYhBnJlj4l2MrI8gw6LU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6811 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Maryam, I have added a small nitpick. Regards, Shibin > -----Original Message----- > From: Maryam Tahhan > Sent: Wednesday, February 14, 2024 5:04 PM > To: ferruh.yigit@amd.com; stephen@networkplumber.org; > lihuisong@huawei.com; fengchengwen@huawei.com; > liuyonglong@huawei.com; Marchand, David > ; Koikkara Reeny, Shibin > ; Loftus, Ciara > Cc: dev@dpdk.org; Tahhan, Maryam ; > stable@dpdk.org > Subject: [v9 1/3] docs: AF_XDP Device Plugin >=20 > Fixup the references to the AF_XDP Device Plugin in the documentation (wa= s > referred to as CNI previously) and document the single netdev limitation = for > deploying an AF_XDP based DPDK pod. Also renames af_xdp_cni.rst to > af_xdp_dp.rst >=20 > Fixes: 7fc6ae50369d ("net/af_xdp: support CNI Integration") > Cc: stable@dpdk.org >=20 > Signed-off-by: Maryam Tahhan > --- > doc/guides/howto/af_xdp_cni.rst | 253 --------------------------- > doc/guides/howto/af_xdp_dp.rst | 299 > ++++++++++++++++++++++++++++++++ > doc/guides/howto/index.rst | 2 +- > doc/guides/nics/af_xdp.rst | 4 +- > 4 files changed, 302 insertions(+), 256 deletions(-) delete mode 100644 > doc/guides/howto/af_xdp_cni.rst create mode 100644 > doc/guides/howto/af_xdp_dp.rst >=20 > diff --git a/doc/guides/howto/af_xdp_cni.rst > b/doc/guides/howto/af_xdp_cni.rst deleted file mode 100644 index > a1a6d5b99c..0000000000 > --- a/doc/guides/howto/af_xdp_cni.rst > +++ /dev/null > @@ -1,253 +0,0 @@ > -.. SPDX-License-Identifier: BSD-3-Clause > - Copyright(c) 2023 Intel Corporation. > - > -Using a CNI with the AF_XDP driver > -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > - > -Introduction > ------------- > - > -CNI, the Container Network Interface, is a technology for configuring - > container network interfaces -and which can be used to setup Kubernetes > networking. > -AF_XDP is a Linux socket Address Family that enables an XDP program -to > redirect packets to a memory buffer in userspace. > - > -This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ > within -a DPDK application using the :doc:`../nics/af_xdp` to connect and= use > these technologies. > - > -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins= - > for-kubernetes > - > - > -Background > ----------- > - > -The standard :doc:`../nics/af_xdp` initialization process involves loadi= ng an > eBPF program -onto the kernel netdev to be used by the PMD. > -This operation requires root or escalated Linux privileges -and thus pre= vents > the PMD from working in an unprivileged container. > -The AF_XDP CNI plugin handles this situation -by providing a device plug= in > that performs the program loading. > - > -At a technical level the CNI opens a Unix Domain Socket and listens for = a > client -to make requests over that socket. > -A DPDK application acting as a client connects and initiates a configura= tion > "handshake". > -The client then receives a file descriptor which points to the XSKMAP - > associated with the loaded eBPF program. > -The XSKMAP is a BPF map of AF_XDP sockets (XSK). > -The client can then proceed with creating an AF_XDP socket -and insertin= g > that socket into the XSKMAP pointed to by the descriptor. > - > -The EAL vdev argument ``use_cni`` is used to indicate that the user wish= es - > to run the PMD in unprivileged mode and to receive the XSKMAP file > descriptor -from the CNI. > -When this flag is set, > -the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag -should be > used when creating the socket -to instruct libbpf not to load the default > libbpf program on the netdev. > -Instead the loading is handled by the CNI. > - > -.. note:: > - > - The Unix Domain Socket file path appear in the end user is > "/tmp/afxdp.sock". > - > - > -Prerequisites > -------------- > - > -Docker and container prerequisites: > - > -* Set up the device plugin > - as described in the instructions for `AF_XDP Plugin for Kubernetes`_. > - > -* The Docker image should contain the libbpf and libxdp libraries, > - which are dependencies for AF_XDP, > - and should include support for the ``ethtool`` command. > - > -* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and > ``CAP_BPF`` > - for AF_XDP along with support for hugepages. > - > -* Increase locked memory limit so containers have enough memory for > packet buffers. > - For example: > - > - .. code-block:: console > - > - cat << EOF | sudo tee > /etc/systemd/system/containerd.service.d/limits.conf > - [Service] > - LimitMEMLOCK=3Dinfinity > - EOF > - > -* dpdk-testpmd application should have AF_XDP feature enabled. > - > - For further information see the docs for the: :doc:`../../nics/af_xdp`= . > - > - > -Example > -------- > - > -Howto run dpdk-testpmd with CNI plugin: > - > -* Clone the CNI plugin > - > - .. code-block:: console > - > - # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.g= it > - > -* Build the CNI plugin > - > - .. code-block:: console > - > - # cd afxdp-plugins-for-kubernetes/ > - # make build > - > - .. note:: > - > - CNI plugin has a dependence on the config.json. > - > - Sample Config.json > - > - .. code-block:: json > - > - { > - "logLevel":"debug", > - "logFile":"afxdp-dp-e2e.log", > - "pools":[ > - { > - "name":"e2e", > - "mode":"primary", > - "timeout":30, > - "ethtoolCmds" : ["-L -device- combined 1"], > - "devices":[ > - { > - "name":"ens785f0" > - } > - ] > - } > - ] > - } > - > - For further reference please use the `config.json`_ > - > - .. _config.json: https://github.com/intel/afxdp-plugins-for- > kubernetes/blob/v0.0.2/test/e2e/config.json > - > -* Create the Network Attachment definition > - > - .. code-block:: console > - > - # kubectl create -f nad.yaml > - > - Sample nad.yml > - > - .. code-block:: yaml > - > - apiVersion: "k8s.cni.cncf.io/v1" > - kind: NetworkAttachmentDefinition > - metadata: > - name: afxdp-e2e-test > - annotations: > - k8s.v1.cni.cncf.io/resourceName: afxdp/e2e > - spec: > - config: '{ > - "cniVersion": "0.3.0", > - "type": "afxdp", > - "mode": "cdq", > - "logFile": "afxdp-cni-e2e.log", > - "logLevel": "debug", > - "ipam": { > - "type": "host-local", > - "subnet": "192.168.1.0/24", > - "rangeStart": "192.168.1.200", > - "rangeEnd": "192.168.1.216", > - "routes": [ > - { "dst": "0.0.0.0/0" } > - ], > - "gateway": "192.168.1.1" > - } > - }' > - > - For further reference please use the `nad.yaml`_ > - > - .. _nad.yaml: https://github.com/intel/afxdp-plugins-for- > kubernetes/blob/v0.0.2/test/e2e/nad.yaml > - > -* Build the Docker image > - > - .. code-block:: console > - > - # docker build -t afxdp-e2e-test -f Dockerfile . > - > - Sample Dockerfile: > - > - .. code-block:: console > - > - FROM ubuntu:20.04 > - RUN apt-get update -y > - RUN apt install build-essential libelf-dev -y > - RUN apt-get install iproute2 acl -y > - RUN apt install python3-pyelftools ethtool -y > - RUN apt install libnuma-dev libjansson-dev libpcap-dev net-tools -y > - RUN apt-get install clang llvm -y > - COPY ./libbpf.tar.gz /tmp > - RUN cd /tmp && tar -xvmf libbpf.tar.gz && cd libbpf/src && > make install > - COPY ./libxdp.tar.gz /tmp > - RUN cd /tmp && tar -xvmf libxdp.tar.gz && cd libxdp && mak= e > install > - > - .. note:: > - > - All the files that need to COPY-ed should be in the same directory = as the > Dockerfile > - > -* Run the Pod > - > - .. code-block:: console > - > - # kubectl create -f pod.yaml > - > - Sample pod.yaml: > - > - .. code-block:: yaml > - > - apiVersion: v1 > - kind: Pod > - metadata: > - name: afxdp-e2e-test > - annotations: > - k8s.v1.cni.cncf.io/networks: afxdp-e2e-test > - spec: > - containers: > - - name: afxdp > - image: afxdp-e2e-test:latest > - imagePullPolicy: Never > - env: > - - name: LD_LIBRARY_PATH > - value: /usr/lib64/:/usr/local/lib/ > - command: ["tail", "-f", "/dev/null"] > - securityContext: > - capabilities: > - add: > - - CAP_NET_RAW > - - CAP_BPF > - resources: > - requests: > - hugepages-2Mi: 2Gi > - memory: 2Gi > - afxdp/e2e: '1' > - limits: > - hugepages-2Mi: 2Gi > - memory: 2Gi > - afxdp/e2e: '1' > - > - For further reference please use the `pod.yaml`_ > - > - .. _pod.yaml: https://github.com/intel/afxdp-plugins-for- > kubernetes/blob/v0.0.2/test/e2e/pod-1c1d.yaml > - > -* Run DPDK with a command like the following: > - > - .. code-block:: console > - > - kubectl exec -i --container -- \ > - //dpdk-testpmd -l 0,1 --no-pci \ > - --vdev=3Dnet_af_xdp0,use_cni=3D1,iface=3D \ > - -- --no-mlockall --in-memory > - > -For further reference please use the `e2e`_ test case in `AF_XDP Plugin = for > Kubernetes`_ > - > - .. _e2e: https://github.com/intel/afxdp-plugins-for- > kubernetes/tree/v0.0.2/test/e2e > diff --git a/doc/guides/howto/af_xdp_dp.rst > b/doc/guides/howto/af_xdp_dp.rst new file mode 100644 index > 0000000000..657fc8d52c > --- /dev/null > +++ b/doc/guides/howto/af_xdp_dp.rst > @@ -0,0 +1,299 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright(c) 2023 Intel Corporation. > + > +Using the AF_XDP driver in Kubernetes > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + > +Introduction > +------------ > + > +Two infrastructure components are needed in order to provision a pod > +that is using the AF_XDP PMD in Kubernetes: > + > +1. AF_XDP Device Plugin (DP). > +2. AF_XDP Container Network Interface (CNI) binary. > + > +Both of these components are available through the `AF_XDP Device > +Plugin for Kubernetes`_ repository. > + > +The AF_XDP DP provisions and advertises networking interfaces to > +Kubernetes, while the CNI configures and plumbs network interfaces for > the Pod. > + > +This document explains how to use the `AF_XDP Device Plugin for > +Kubernetes`_ with a DPDK application using the :doc:`../nics/af_xdp`. > + > +.. _AF_XDP Device Plugin for Kubernetes: > +https://github.com/intel/afxdp-plugins-for-kubernetes > + > +Background > +---------- > + > +The standard :doc:`../nics/af_xdp` initialization process involves > +loading an eBPF program onto the kernel netdev to be used by the PMD. > +This operation requires root or escalated Linux privileges and thus > +prevents the PMD from working in an unprivileged container. > +The AF_XDP Device Plugin handles this situation by managing the eBPF > +program(s) on behalf of the Pod, outside of the pod context. > + > +At a technical level the AF_XDP Device Plugin opens a Unix Domain > +Socket Small nitpick: added the abbreviation UDS as it used in the below sections= =20 and listens for a client to make requests over that socket. > +A DPDK application acting as a client connects and initiates a configura= tion > "handshake". > +After some validation on the Device Plugin side, the client receives a > +file descriptor which points to the XSKMAP associated with the loaded eB= PF > program. > +The XSKMAP is an eBPF map of AF_XDP sockets (XSK). > +The client can then proceed with creating an AF_XDP socket and > +inserting that socket into the XSKMAP pointed to by the descriptor. > + > +The EAL vdev argument ``use_cni`` is used to indicate that the user > +wishes to run the PMD in unprivileged mode and to receive the XSKMAP > +file descriptor from the CNI. > +When this flag is set, > +the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag should be > used > +when creating the socket to instruct libbpf not to load the default > +libbpf program on the netdev. > +Instead the loading is handled by the AF_XDP Device Plugin. > + > +Limitations > +----------- > + > +For DPDK versions <=3D v23.11 the Unix Domain Socket file path appears i= n > +the pod at "/tmp/afxdp.sock". The handshake implementation in the > +AF_XDP PMD is only compatible with the AF_XDP Device Plugin up to > +commit id `38317c2`_ and the pod is limited to a single netdev. > + > +.. note:: > + > + DPDK AF_XDP PMD <=3D v23.11 will not work with the latest version of= the > + AF_XDP Device Plugin. > + > +The issue is if a single pod requests different devices from different > +pools it results in multiple UDS servers serving the pod with the > +container using only a single mount point for their UDS as > +``/tmp/afxdp.sock``. This means that at best one device might be able > +to complete the handshake. This has been fixed in the AF_XDP Device > +Plugin so that the mount point in the pods for the UDS appear at > +``/tmp/afxdp_dp//afxdp.sock``. Later versions of DPDK fix this > hardcoded path in the PMD alongside the `use_cni` parameter. Small nitpick: `use_cni` should be ``use_cni`` > + > +.. _38317c2: > +https://github.com/intel/afxdp-plugins-for- > kubernetes/commit/38317c256b > +5c7dfb39e013a0f76010c2ded03669 > + > + > +Prerequisites > +------------- > + > +Device Plugin and DPDK container prerequisites: > + > +* Create a DPDK container image. > + > +* Set up the device plugin and prepare the Pod Spec as described in > + the instructions for `AF_XDP Device Plugin for Kubernetes`_. > + > +* The Docker image should contain the libbpf and libxdp libraries, > + which are dependencies for AF_XDP, > + and should include support for the ``ethtool`` command. > + > +* The Pod should have enabled the capabilities ``CAP_NET_RAW`` for > + AF_XDP socket creation, ``IPC_LOCK`` for umem creation and > + ``CAP_BPF`` (for Kernel < 5.19) along with support for hugepages. > + > + .. note:: > + > + For Kernel versions < 5.19, all BPF sys calls required CAP_BPF, to a= ccess > maps shared > + between the eBFP program and the userspace program. Kernels >=3D 5.1= 9, > only requires CAP_BPF > + for map creation (BPF_MAP_CREATE) and loading programs > (BPF_PROG_LOAD). > + > +* Increase locked memory limit so containers have enough memory for > packet buffers. > + For example: > + > + .. code-block:: console > + > + cat << EOF | sudo tee > /etc/systemd/system/containerd.service.d/limits.conf > + [Service] > + LimitMEMLOCK=3Dinfinity > + EOF > + > +* dpdk-testpmd application should have AF_XDP feature enabled. > + > + For further information see the docs for the: :doc:`../../nics/af_xdp`= . > + > + > +Example > +------- > + > +Build a DPDK container image (using Docker) > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +1. Create a Dockerfile (should be placed in top level DPDK directory): > + > + .. code-block:: console > + > + FROM fedora:38 > + > + # Setup container to build DPDK applications > + RUN dnf -y upgrade && dnf -y install \ > + libbsd-devel \ > + numactl-libs \ > + libbpf-devel \ > + libbpf \ > + meson \ > + ninja-build \ > + libxdp-devel \ > + libxdp \ > + numactl-devel \ > + python3-pyelftools \ > + python38 \ > + iproute > + RUN dnf groupinstall -y 'Development Tools' > + > + # Create DPDK dir and copy over sources > + # Create DPDK dir and copy over sources > + COPY ./ /dpdk > + WORKDIR /dpdk > + > + # Build DPDK > + RUN meson setup build > + RUN ninja -C build > + > +2. Build a DPDK container image (using Docker) > + > + .. code-block:: console > + > + # docker build -t dpdk -f Dockerfile > + > +Run dpdk-testpmd with the AF_XDP Device Plugin + CNI > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +* Clone the AF_XDP Device plugin and CNI > + > + .. code-block:: console > + > + # git clone > + https://github.com/intel/afxdp-plugins-for-kubernetes.git > + > + .. note:: > + > + Ensure you have the AF_XDP Device Plugin + CNI prerequisites install= ed. > + > +* Build the AF_XDP Device plugin and CNI > + > + .. code-block:: console > + > + # cd afxdp-plugins-for-kubernetes/ > + # make image > + > +* Make sure to modify the image used by the `daemonset.yml`_ file in > +the deployments directory with > + the following configuration: > + > + .. _daemonset.yml : > + https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/deploy > + ments/daemonset.yml > + > + .. code-block:: yaml > + > + image: afxdp-device-plugin:latest > + > + .. note:: > + > + This will select the AF_XDP DP image that was built locally. Detaile= d > configuration > + options can be found in the AF_XDP Device Plugin `readme`_ . > + > + .. _readme: > + https://github.com/intel/afxdp-plugins-for-kubernetes#readme > + > +* Deploy the AF_XDP Device Plugin and CNI > + > + .. code-block:: console > + > + # kubectl create -f deployments/daemonset.yml > + > +* Create the Network Attachment definition > + > + .. code-block:: console > + > + # kubectl create -f nad.yaml > + > + Sample nad.yml > + > + .. code-block:: yaml > + > + apiVersion: "k8s.cni.cncf.io/v1" > + kind: NetworkAttachmentDefinition > + metadata: > + name: afxdp-network > + annotations: > + k8s.v1.cni.cncf.io/resourceName: afxdp/myPool > + spec: > + config: '{ > + "cniVersion": "0.3.0", > + "type": "afxdp", > + "mode": "primary", > + "logFile": "afxdp-cni.log", > + "logLevel": "debug", > + "ethtoolCmds" : ["-N -device- rx-flow-hash udp4 fn", > + "-N -device- flow-type udp4 dst-port 2152 act= ion 22" > + ], > + "ipam": { > + "type": "host-local", > + "subnet": "192.168.1.0/24", > + "rangeStart": "192.168.1.200", > + "rangeEnd": "192.168.1.220", > + "routes": [ > + { "dst": "0.0.0.0/0" } > + ], > + "gateway": "192.168.1.1" > + } > + }' > + > + For further reference please use the example provided by the AF_XDP > + DP `nad.yaml`_ > + > + .. _nad.yaml: > + https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/exampl > + es/network-attachment-definition.yaml > + > +* Run the Pod > + > + .. code-block:: console > + > + # kubectl create -f pod.yaml > + > + Sample pod.yaml: > + > + .. code-block:: yaml > + > + apiVersion: v1 > + kind: Pod > + metadata: > + name: dpdk > + annotations: > + k8s.v1.cni.cncf.io/networks: afxdp-network > + spec: > + containers: > + - name: testpmd > + image: dpdk:latest > + command: ["tail", "-f", "/dev/null"] > + securityContext: > + capabilities: > + add: > + - NET_RAW > + - IPC_LOCK > + resources: > + requests: > + afxdp/myPool: '1' > + limits: > + hugepages-1Gi: 2Gi > + cpu: 2 > + memory: 256Mi > + afxdp/myPool: '1' > + volumeMounts: > + - name: hugepages > + mountPath: /dev/hugepages > + volumes: > + - name: hugepages > + emptyDir: > + medium: HugePages > + > + For further reference please use the `pod.yaml`_ > + > + .. _pod.yaml: > + https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/exampl > + es/pod-spec.yaml > + > +* Run DPDK with a command like the following: > + > + .. code-block:: console > + > + kubectl exec -i --container -- \ > + //dpdk-testpmd -l 0,1 --no-pci \ > + --vdev=3Dnet_af_xdp0,use_cni=3D1,iface=3D \ > + --no-mlockall --in-memory \ > + -- -i --a --nb-cores=3D2 --rxq=3D1 --txq=3D1 > + --forward-mode=3Dmacswap; > diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst inde= x > 71a3381c36..a7692e8a97 100644 > --- a/doc/guides/howto/index.rst > +++ b/doc/guides/howto/index.rst > @@ -8,7 +8,7 @@ HowTo Guides > :maxdepth: 2 > :numbered: >=20 > - af_xdp_cni > + af_xdp_dp > lm_bond_virtio_sriov > lm_virtio_vhost_user > flow_bifurcation > diff --git a/doc/guides/nics/af_xdp.rst b/doc/guides/nics/af_xdp.rst inde= x > 1932525d4d..4dd9c73742 100644 > --- a/doc/guides/nics/af_xdp.rst > +++ b/doc/guides/nics/af_xdp.rst > @@ -155,9 +155,9 @@ use_cni > ~~~~~~~ >=20 > The EAL vdev argument ``use_cni`` is used to indicate that the user wish= es to > -enable the `AF_XDP Plugin for Kubernetes`_ within a DPDK application. > +enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK > application/pod. >=20 > -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins= - > for-kubernetes > +.. _AF_XDP Device Plugin for Kubernetes: > +https://github.com/intel/afxdp-plugins-for-kubernetes >=20 > .. code-block:: console >=20 > -- > 2.41.0