From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3D8F345610 for ; Fri, 12 Jul 2024 12:47:27 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 37950402E5; Fri, 12 Jul 2024 12:47:27 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2062.outbound.protection.outlook.com [40.107.92.62]) by mails.dpdk.org (Postfix) with ESMTP id DCFE740261 for ; Fri, 12 Jul 2024 12:47:25 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=o0e7cUd78MNqUM8c3pRBG/0BnQNJk2jyTHmrd7djFaBJL7tMOBMdnYh+7d9h66ijAimHjVVkCOqlKYN/EMOdR7PzsOYVZHxyC5mC9i7Rvpl4FZg/wUv2lD2LYg18R2VvOZ/Y1bYbMYMZPtqk1gtXPiY0IfIE1dJQTzt+K8eruGKSIfmZjduSDRFIA9TIYaEzQ/YGEbFAxvR+WdkHbHK/+fMX3RYTmRiAUbrSIRq1wJC+W3YiW1QMaib9w9/bh0Hr3XGQZNTfBLghYfv2w9oLSwZxLJrdcFVxF8xvR4K4UQOgkoCEBaG0yOkqhD4i511AB051aTe+APrQDXMxXpE9eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FgRPytKO8uCmpzZ9dVE0UQv0Rf4PVsDSbjhL4WOcRDQ=; b=k9jK+MboG30yUJRXw3R+5bZqB2nxrvPbXCR8XX/ktuBUSM2lEPjwF+RIX7fzK+0igClJadT56OiNWmpW9kxOLSUQ03gJSNf/3SBhMuuMUKh1Kzarmx+FfIpSPM8qXe7W9SdOMpE+U+8Aa+/ksFWuUOviBoMVyG/J1fjP5zuisFDhvvJLGXoUol1b/+PoqxKjR34KYnqJ+NbHOTVKja6focvVNEN60YBRmfPCXLPSV3sjzkd+T4fpttQ7zhkE113gyD9SnP3XhC7xHQccTmjX69pyJnKWhiiDgDmxOhqGFfGBYGqYulp2TziunJGERiSbhbJ5ozbgaja9JCRL+50/qw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FgRPytKO8uCmpzZ9dVE0UQv0Rf4PVsDSbjhL4WOcRDQ=; b=eoTEyt1dEKQFfygrXvb8aZUBg5sslpqndhlbaJsWGrVpLgYV1XFpIqcb1aCOJgP7OXLq0qNJAOsrzzpAh5UfFoyyfaovCzcnLx7dPgWTmFif4GddjAwHSaN8n3e1uJcM1tidCC5mVHGs1+r0qTw4xYuFCfg8x8RgiBQu9+Cg6jO9c5rPw6NctG9xJbxHqScYxTPyCCpc/x1EjX/GA9MejTS5ZC86fvVYJOnOSMQJ5GHxNn6Sy3uQcL7vFUMouUdhOlogHKVxInhhm8NlRHCv4EhUjw3xANfV05J3LLQKbjr1w7kGMZ5m1rX/JW2h8Wp0hVNEvxKCd6w4BqvU3Y3JiQ== Received: from DM6PR02CA0141.namprd02.prod.outlook.com (2603:10b6:5:332::8) by DS0PR12MB7510.namprd12.prod.outlook.com (2603:10b6:8:132::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.39; Fri, 12 Jul 2024 10:47:22 +0000 Received: from DS1PEPF00017098.namprd05.prod.outlook.com (2603:10b6:5:332:cafe::26) by DM6PR02CA0141.outlook.office365.com (2603:10b6:5:332::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7762.24 via Frontend Transport; Fri, 12 Jul 2024 10:47:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS1PEPF00017098.mail.protection.outlook.com (10.167.18.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7762.17 via Frontend Transport; Fri, 12 Jul 2024 10:47:22 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 12 Jul 2024 03:47:13 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 12 Jul 2024 03:47:11 -0700 From: Xueming Li To: Maryam Tahhan CC: Ciara Loftus , dpdk stable Subject: patch 'doc: fix AF_XDP device plugin howto' has been queued to stable release 23.11.2 Date: Fri, 12 Jul 2024 18:44:02 +0800 Message-ID: <20240712104528.308638-17-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240712104528.308638-1-xuemingl@nvidia.com> References: <20240712104528.308638-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF00017098:EE_|DS0PR12MB7510:EE_ X-MS-Office365-Filtering-Correlation-Id: 12e3e314-4e69-47f1-7600-08dca26002bc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|36860700013|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?BJevn4gwF846qZ//ykLYRi9HGXf64cRDa9gHRrQvMUBeO73/PxxubuDC5tz3?= =?us-ascii?Q?Z9VXan/s64XXa2pCOnh1JJf27k188Zq6Ter9dVabkJRO5d7pj8R+gfuTRuTG?= =?us-ascii?Q?eVbUUt18/nXALgT/nOhAKHrwAtYdH7nrCv5J4eD3kavYSKmgtcEd8lf/jbMJ?= =?us-ascii?Q?FUTzugzVMDrGQ15HyCp9X4wODHToeaHkFI2NnttuN+BkaVr3QeS1o8ZRmIpf?= =?us-ascii?Q?m5s3PBqK8st/Xtg0FvBXB6tv7F5vqwrrRkowaDWE/dIbdhuyIcEvNzUJonMc?= =?us-ascii?Q?dXou3JhbFti+nXolWzmQpwBhdy/oaYhRvJmAOHaIryCahtuEYzXNnGfF9THb?= =?us-ascii?Q?VjnUbrgOQI02Kn8UxgIDamWv07qfacVdkfhvcXDvLEK9CKvWg8SSjSTGCFw6?= =?us-ascii?Q?EKx767gdrgCJHtEy6XPJT+IIAwCqp50mHtN7Ub5FsbtCz5JWAToGz7bkRJDy?= =?us-ascii?Q?q3DC57P2767ROsjFEl5Is+g1sfOXpv5WGkO+jHMhkuSne77spnLA6FMjWRZh?= =?us-ascii?Q?baG/kxeKrx56hQjpBUbDLAr/XeJs6ITL1WtQ7cnFhWuHokm2tnJqud9Dyzua?= =?us-ascii?Q?iKjF2S+zp/pT2yCNbUjgDsjowiNkFtNZ6ujmWEJU1wtF82LItpjJ0GyoO8l6?= =?us-ascii?Q?6RSeiHeaJuHHaTSj+JNODAzV9rHUYTlk5rxyr6/oKik3JKG62459QjOAjJAP?= =?us-ascii?Q?vttgzii5Fco6obRGWVIAIIfNAbcmlZhp9IalCfJbkzP4HKEc5pUGUrdpJM9S?= =?us-ascii?Q?oOJY297hFFiXf56qUDg4N6DEuakQm7onjk567gd3yh6eVAPJq/SK53kvcayZ?= =?us-ascii?Q?8xUBxvOe9qlLicwNIuUmEIo0i5qtr47rgeot0+3u2XnH7+4vUthBph5JY4Os?= =?us-ascii?Q?7YX8lfmJY9U0VEdI9PrYjkhKEjrKhahdJ6+B2hnN7u/Bpw56/6ucmoWqNRMF?= =?us-ascii?Q?fpShgLKD9n3SGLo5wtpgztb1eNFm6uwMV6BEVE4hCCUgOl7ghq7vdn0BL74T?= =?us-ascii?Q?B1dqTbEBcZ4WkLUYXpN42iErB+pBcv5D54NKkglnQwfrQtwcpT7iMMUqJHMf?= =?us-ascii?Q?pw9LJYiSC0eIQ1CYYLgPESyFYO5GkfRfxpOBHB8PQOscMocJbYDFmqKjhsTm?= =?us-ascii?Q?tfSK6EoK5mwBqxOvdaIn0h+ZSg/3nj8w4/fyq+XglBRiwKVSlMo1JAmP/zSO?= =?us-ascii?Q?dFmkDp2TWic5SCN+7KoKAN4WNA2qbiXKrCzBm4Npbe0aq4BRIy5VJiEaSjuy?= =?us-ascii?Q?zRevRybsMDH1dFdoaeyApTNjQz9+JaT4eWmttmDRMAdUWxqFlWF38P1LlTxq?= =?us-ascii?Q?GVPYmRLMTw/jsMtrR1EGPd4mZ7AHxyPpDRVwEET3725jW9suXXqSM6cWU4ra?= =?us-ascii?Q?IIHgTmLc57gA+N1ngJg/9XnwPXwJFK6QAI2/UDrAppCzNNz08UBbDZv2X5jx?= =?us-ascii?Q?JfMurDeXTud6kx3zRb+gHv/cunXoIp3S?= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2024 10:47:22.4983 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 12e3e314-4e69-47f1-7600-08dca26002bc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF00017098.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7510 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 23.11.2 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 07/14/24. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://git.dpdk.org/dpdk-stable/log/?h=23.11-staging This queued commit can be viewed at: https://git.dpdk.org/dpdk-stable/commit/?h=23.11-staging&id=190a4d4844aeca2c94e4b0f49065057928bf202e Thanks. Xueming Li --- >From 190a4d4844aeca2c94e4b0f49065057928bf202e Mon Sep 17 00:00:00 2001 From: Maryam Tahhan Date: Mon, 8 Apr 2024 09:09:20 -0400 Subject: [PATCH] doc: fix AF_XDP device plugin howto Cc: Xueming Li [ upstream commit 696982d5efd1ac40576c076b4358a144181d4a0e ] Fixup the references to the AF_XDP Device Plugin in the documentation (was referred to as CNI previously) and document the single netdev limitation for deploying an AF_XDP based DPDK pod. Also renames af_xdp_cni.rst to af_xdp_dp.rst Fixes: 7fc6ae50369d ("net/af_xdp: support CNI Integration") Signed-off-by: Maryam Tahhan Acked-by: Ciara Loftus --- .mailmap | 2 +- doc/guides/howto/af_xdp_cni.rst | 253 -------------------------- doc/guides/howto/af_xdp_dp.rst | 306 ++++++++++++++++++++++++++++++++ doc/guides/howto/index.rst | 2 +- doc/guides/nics/af_xdp.rst | 4 +- 5 files changed, 310 insertions(+), 257 deletions(-) delete mode 100644 doc/guides/howto/af_xdp_cni.rst create mode 100644 doc/guides/howto/af_xdp_dp.rst diff --git a/.mailmap b/.mailmap index f3b429b747..c6bf385a59 100644 --- a/.mailmap +++ b/.mailmap @@ -899,7 +899,7 @@ Martin Klozik Martin Spinler Martin Weiser Martyna Szapar-Mudlaw -Maryam Tahhan +Maryam Tahhan Masoud Hasanifard Masoumeh Farhadi Nia Matan Azrad diff --git a/doc/guides/howto/af_xdp_cni.rst b/doc/guides/howto/af_xdp_cni.rst deleted file mode 100644 index a1a6d5b99c..0000000000 --- a/doc/guides/howto/af_xdp_cni.rst +++ /dev/null @@ -1,253 +0,0 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2023 Intel Corporation. - -Using a CNI with the AF_XDP driver -================================== - -Introduction ------------- - -CNI, the Container Network Interface, is a technology for configuring -container network interfaces -and which can be used to setup Kubernetes networking. -AF_XDP is a Linux socket Address Family that enables an XDP program -to redirect packets to a memory buffer in userspace. - -This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within -a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies. - -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes - - -Background ----------- - -The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program -onto the kernel netdev to be used by the PMD. -This operation requires root or escalated Linux privileges -and thus prevents the PMD from working in an unprivileged container. -The AF_XDP CNI plugin handles this situation -by providing a device plugin that performs the program loading. - -At a technical level the CNI opens a Unix Domain Socket and listens for a client -to make requests over that socket. -A DPDK application acting as a client connects and initiates a configuration "handshake". -The client then receives a file descriptor which points to the XSKMAP -associated with the loaded eBPF program. -The XSKMAP is a BPF map of AF_XDP sockets (XSK). -The client can then proceed with creating an AF_XDP socket -and inserting that socket into the XSKMAP pointed to by the descriptor. - -The EAL vdev argument ``use_cni`` is used to indicate that the user wishes -to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor -from the CNI. -When this flag is set, -the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag -should be used when creating the socket -to instruct libbpf not to load the default libbpf program on the netdev. -Instead the loading is handled by the CNI. - -.. note:: - - The Unix Domain Socket file path appear in the end user is "/tmp/afxdp.sock". - - -Prerequisites -------------- - -Docker and container prerequisites: - -* Set up the device plugin - as described in the instructions for `AF_XDP Plugin for Kubernetes`_. - -* The Docker image should contain the libbpf and libxdp libraries, - which are dependencies for AF_XDP, - and should include support for the ``ethtool`` command. - -* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and ``CAP_BPF`` - for AF_XDP along with support for hugepages. - -* Increase locked memory limit so containers have enough memory for packet buffers. - For example: - - .. code-block:: console - - cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf - [Service] - LimitMEMLOCK=infinity - EOF - -* dpdk-testpmd application should have AF_XDP feature enabled. - - For further information see the docs for the: :doc:`../../nics/af_xdp`. - - -Example -------- - -Howto run dpdk-testpmd with CNI plugin: - -* Clone the CNI plugin - - .. code-block:: console - - # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git - -* Build the CNI plugin - - .. code-block:: console - - # cd afxdp-plugins-for-kubernetes/ - # make build - - .. note:: - - CNI plugin has a dependence on the config.json. - - Sample Config.json - - .. code-block:: json - - { - "logLevel":"debug", - "logFile":"afxdp-dp-e2e.log", - "pools":[ - { - "name":"e2e", - "mode":"primary", - "timeout":30, - "ethtoolCmds" : ["-L -device- combined 1"], - "devices":[ - { - "name":"ens785f0" - } - ] - } - ] - } - - For further reference please use the `config.json`_ - - .. _config.json: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/config.json - -* Create the Network Attachment definition - - .. code-block:: console - - # kubectl create -f nad.yaml - - Sample nad.yml - - .. code-block:: yaml - - apiVersion: "k8s.cni.cncf.io/v1" - kind: NetworkAttachmentDefinition - metadata: - name: afxdp-e2e-test - annotations: - k8s.v1.cni.cncf.io/resourceName: afxdp/e2e - spec: - config: '{ - "cniVersion": "0.3.0", - "type": "afxdp", - "mode": "cdq", - "logFile": "afxdp-cni-e2e.log", - "logLevel": "debug", - "ipam": { - "type": "host-local", - "subnet": "192.168.1.0/24", - "rangeStart": "192.168.1.200", - "rangeEnd": "192.168.1.216", - "routes": [ - { "dst": "0.0.0.0/0" } - ], - "gateway": "192.168.1.1" - } - }' - - For further reference please use the `nad.yaml`_ - - .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/nad.yaml - -* Build the Docker image - - .. code-block:: console - - # docker build -t afxdp-e2e-test -f Dockerfile . - - Sample Dockerfile: - - .. code-block:: console - - FROM ubuntu:20.04 - RUN apt-get update -y - RUN apt install build-essential libelf-dev -y - RUN apt-get install iproute2 acl -y - RUN apt install python3-pyelftools ethtool -y - RUN apt install libnuma-dev libjansson-dev libpcap-dev net-tools -y - RUN apt-get install clang llvm -y - COPY ./libbpf.tar.gz /tmp - RUN cd /tmp && tar -xvmf libbpf.tar.gz && cd libbpf/src && make install - COPY ./libxdp.tar.gz /tmp - RUN cd /tmp && tar -xvmf libxdp.tar.gz && cd libxdp && make install - - .. note:: - - All the files that need to COPY-ed should be in the same directory as the Dockerfile - -* Run the Pod - - .. code-block:: console - - # kubectl create -f pod.yaml - - Sample pod.yaml: - - .. code-block:: yaml - - apiVersion: v1 - kind: Pod - metadata: - name: afxdp-e2e-test - annotations: - k8s.v1.cni.cncf.io/networks: afxdp-e2e-test - spec: - containers: - - name: afxdp - image: afxdp-e2e-test:latest - imagePullPolicy: Never - env: - - name: LD_LIBRARY_PATH - value: /usr/lib64/:/usr/local/lib/ - command: ["tail", "-f", "/dev/null"] - securityContext: - capabilities: - add: - - CAP_NET_RAW - - CAP_BPF - resources: - requests: - hugepages-2Mi: 2Gi - memory: 2Gi - afxdp/e2e: '1' - limits: - hugepages-2Mi: 2Gi - memory: 2Gi - afxdp/e2e: '1' - - For further reference please use the `pod.yaml`_ - - .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/pod-1c1d.yaml - -* Run DPDK with a command like the following: - - .. code-block:: console - - kubectl exec -i --container -- \ - //dpdk-testpmd -l 0,1 --no-pci \ - --vdev=net_af_xdp0,use_cni=1,iface= \ - -- --no-mlockall --in-memory - -For further reference please use the `e2e`_ test case in `AF_XDP Plugin for Kubernetes`_ - - .. _e2e: https://github.com/intel/afxdp-plugins-for-kubernetes/tree/v0.0.2/test/e2e diff --git a/doc/guides/howto/af_xdp_dp.rst b/doc/guides/howto/af_xdp_dp.rst new file mode 100644 index 0000000000..2f51b37f20 --- /dev/null +++ b/doc/guides/howto/af_xdp_dp.rst @@ -0,0 +1,306 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2023 Intel Corporation. + +Using the AF_XDP driver in Kubernetes +===================================== + +Introduction +------------ + +Two infrastructure components are needed in order to provision a pod +that is using the AF_XDP PMD in Kubernetes: + +1. AF_XDP Device Plugin (DP). +2. AF_XDP Container Network Interface (CNI) binary. + +Both of these components are available through +the `AF_XDP Device Plugin for Kubernetes`_ repository. + +The AF_XDP DP provisions and advertises networking interfaces to Kubernetes, +while the CNI configures and plumbs network interfaces for the Pod. + +This document explains how to use the `AF_XDP Device Plugin for Kubernetes`_ +with a DPDK application using the :doc:`../nics/af_xdp`. + +.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes + + +Background +---------- + +The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program +onto the kernel netdev to be used by the PMD. +This operation requires root or escalated Linux privileges +and thus prevents the PMD from working in an unprivileged container. +The AF_XDP Device Plugin handles this situation +by managing the eBPF program(s) on behalf of the Pod, outside of the pod context. + +At a technical level the AF_XDP Device Plugin opens a Unix Domain Socket (UDS) +and listens for a client to make requests over that socket. +A DPDK application acting as a client connects and initiates a configuration "handshake". +After some validation on the Device Plugin side, +the client receives a file descriptor which points to the XSKMAP +associated with the loaded eBPF program. +The XSKMAP is an eBPF map of AF_XDP sockets (XSK). +The client can then proceed with creating an AF_XDP socket +and inserting that socket into the XSKMAP pointed to by the descriptor. + +The EAL vdev argument ``use_cni`` is used to indicate that the user wishes +to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor +from the CNI. +When this flag is set, +the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag +should be used when creating the socket +to instruct libbpf not to load the default libbpf program on the netdev. +Instead the loading is handled by the AF_XDP Device Plugin. + + +Limitations +----------- + +For DPDK versions <= v23.11 the Unix Domain Socket file path +appears in the pod at "/tmp/afxdp.sock". +The handshake implementation in the AF_XDP PMD +is only compatible with the AF_XDP Device Plugin up to commit id `38317c2`_ +and the pod is limited to a single netdev. + +.. note:: + + DPDK AF_XDP PMD <= v23.11 will not work with the latest version + of the AF_XDP Device Plugin. + +The issue is if a single pod requests different devices from different pools, +it results in multiple UDS servers serving the pod +with the container using only a single mount point for their UDS as ``/tmp/afxdp.sock``. +This means that at best one device might be able to complete the handshake. +This has been fixed in the AF_XDP Device Plugin so that the mount point in the pods +for the UDS appear at ``/tmp/afxdp_dp//afxdp.sock``. +Later versions of DPDK fix this hardcoded path in the PMD +alongside the ``use_cni`` parameter. + +.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669 + + +Prerequisites +------------- + +Device Plugin and DPDK container prerequisites: + +* Create a DPDK container image. + +* Set up the device plugin and prepare the Pod Spec as described in + the instructions for `AF_XDP Device Plugin for Kubernetes`_. + +* The Docker image should contain the libbpf and libxdp libraries, + which are dependencies for AF_XDP, + and should include support for the ``ethtool`` command. + +* The Pod should have enabled the capabilities + ``CAP_NET_RAW`` for AF_XDP socket creation, + ``IPC_LOCK`` for umem creation and + ``CAP_BPF`` (for Kernel < 5.19) along with support for hugepages. + + .. note:: + + For Kernel versions < 5.19, all BPF sys calls required CAP_BPF, + to access maps shared between the eBFP program and the userspace program. + Kernels >= 5.19, only requires CAP_BPF for map creation (BPF_MAP_CREATE) + and loading programs (BPF_PROG_LOAD). + +* Increase locked memory limit so containers have enough memory for packet buffers. + For example: + + .. code-block:: console + + cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf + [Service] + LimitMEMLOCK=infinity + EOF + +* dpdk-testpmd application should have AF_XDP feature enabled. + + For further information see the docs for the: :doc:`../../nics/af_xdp`. + + +Example +------- + +Build a DPDK container image (using Docker) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Create a Dockerfile (should be placed in top level DPDK directory): + + .. code-block:: console + + FROM fedora:38 + + # Setup container to build DPDK applications + RUN dnf -y upgrade && dnf -y install \ + libbsd-devel \ + numactl-libs \ + libbpf-devel \ + libbpf \ + meson \ + ninja-build \ + libxdp-devel \ + libxdp \ + numactl-devel \ + python3-pyelftools \ + python38 \ + iproute + RUN dnf groupinstall -y 'Development Tools' + + # Create DPDK dir and copy over sources + # Create DPDK dir and copy over sources + COPY ./ /dpdk + WORKDIR /dpdk + + # Build DPDK + RUN meson setup build + RUN ninja -C build + +2. Build a DPDK container image (using Docker) + + .. code-block:: console + + # docker build -t dpdk -f Dockerfile + +Run dpdk-testpmd with the AF_XDP Device Plugin + CNI +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Clone the AF_XDP Device plugin and CNI + + .. code-block:: console + + # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git + + .. note:: + + Ensure you have the AF_XDP Device Plugin + CNI prerequisites installed. + +* Build the AF_XDP Device plugin and CNI + + .. code-block:: console + + # cd afxdp-plugins-for-kubernetes/ + # make image + +* Make sure to modify the image used by the `daemonset.yml`_ file + in the deployments directory with the following configuration: + + .. _daemonset.yml : https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/deployments/daemonset.yml + + .. code-block:: yaml + + image: afxdp-device-plugin:latest + + .. note:: + + This will select the AF_XDP DP image that was built locally. + Detailed configuration options can be found in the AF_XDP Device Plugin `readme`_ . + + .. _readme: https://github.com/intel/afxdp-plugins-for-kubernetes#readme + +* Deploy the AF_XDP Device Plugin and CNI + + .. code-block:: console + + # kubectl create -f deployments/daemonset.yml + +* Create the Network Attachment definition + + .. code-block:: console + + # kubectl create -f nad.yaml + + Sample nad.yml + + .. code-block:: yaml + + apiVersion: "k8s.cni.cncf.io/v1" + kind: NetworkAttachmentDefinition + metadata: + name: afxdp-network + annotations: + k8s.v1.cni.cncf.io/resourceName: afxdp/myPool + spec: + config: '{ + "cniVersion": "0.3.0", + "type": "afxdp", + "mode": "primary", + "logFile": "afxdp-cni.log", + "logLevel": "debug", + "ethtoolCmds" : ["-N -device- rx-flow-hash udp4 fn", + "-N -device- flow-type udp4 dst-port 2152 action 22" + ], + "ipam": { + "type": "host-local", + "subnet": "192.168.1.0/24", + "rangeStart": "192.168.1.200", + "rangeEnd": "192.168.1.220", + "routes": [ + { "dst": "0.0.0.0/0" } + ], + "gateway": "192.168.1.1" + } + }' + + For further reference please use the example provided by the AF_XDP DP `nad.yaml`_ + + .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/network-attachment-definition.yaml + +* Run the Pod + + .. code-block:: console + + # kubectl create -f pod.yaml + + Sample pod.yaml: + + .. code-block:: yaml + + apiVersion: v1 + kind: Pod + metadata: + name: dpdk + annotations: + k8s.v1.cni.cncf.io/networks: afxdp-network + spec: + containers: + - name: testpmd + image: dpdk:latest + command: ["tail", "-f", "/dev/null"] + securityContext: + capabilities: + add: + - NET_RAW + - IPC_LOCK + resources: + requests: + afxdp/myPool: '1' + limits: + hugepages-1Gi: 2Gi + cpu: 2 + memory: 256Mi + afxdp/myPool: '1' + volumeMounts: + - name: hugepages + mountPath: /dev/hugepages + volumes: + - name: hugepages + emptyDir: + medium: HugePages + + For further reference please use the `pod.yaml`_ + + .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/pod-spec.yaml + +* Run DPDK with a command like the following: + + .. code-block:: console + + kubectl exec -i --container -- \ + //dpdk-testpmd -l 0,1 --no-pci \ + --vdev=net_af_xdp0,use_cni=1,iface= \ + --no-mlockall --in-memory \ + -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst index 71a3381c36..a7692e8a97 100644 --- a/doc/guides/howto/index.rst +++ b/doc/guides/howto/index.rst @@ -8,7 +8,7 @@ HowTo Guides :maxdepth: 2 :numbered: - af_xdp_cni + af_xdp_dp lm_bond_virtio_sriov lm_virtio_vhost_user flow_bifurcation diff --git a/doc/guides/nics/af_xdp.rst b/doc/guides/nics/af_xdp.rst index 1932525d4d..4dd9c73742 100644 --- a/doc/guides/nics/af_xdp.rst +++ b/doc/guides/nics/af_xdp.rst @@ -155,9 +155,9 @@ use_cni ~~~~~~~ The EAL vdev argument ``use_cni`` is used to indicate that the user wishes to -enable the `AF_XDP Plugin for Kubernetes`_ within a DPDK application. +enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod. -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes +.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes .. code-block:: console -- 2.34.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2024-07-12 18:40:15.022579617 +0800 +++ 0016-doc-fix-AF_XDP-device-plugin-howto.patch 2024-07-12 18:40:13.946594246 +0800 @@ -1 +1 @@ -From 696982d5efd1ac40576c076b4358a144181d4a0e Mon Sep 17 00:00:00 2001 +From 190a4d4844aeca2c94e4b0f49065057928bf202e Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit 696982d5efd1ac40576c076b4358a144181d4a0e ] @@ -13 +15,0 @@ -Cc: stable@dpdk.org @@ -28 +30 @@ -index 237698aa52..de326949c4 100644 +index f3b429b747..c6bf385a59 100644 @@ -31 +33 @@ -@@ -907,7 +907,7 @@ Martin Klozik +@@ -899,7 +899,7 @@ Martin Klozik