[v12 1/3] docs: AF_XDP Device Plugin

[Maryam Tahhan <mtahhan@redhat.com>]

Fixup the references to the AF_XDP Device Plugin in
the documentation (was referred to as CNI previously)
and document the single netdev limitation for deploying
an AF_XDP based DPDK pod. Also renames af_xdp_cni.rst to
af_xdp_dp.rst

Fixes: 7fc6ae50369d ("net/af_xdp: support CNI Integration")
Cc: stable@dpdk.org

Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>
---
 doc/guides/howto/af_xdp_cni.rst | 253 ---------------------------
 doc/guides/howto/af_xdp_dp.rst  | 299 ++++++++++++++++++++++++++++++++
 doc/guides/howto/index.rst      |   2 +-
 doc/guides/nics/af_xdp.rst      |   4 +-
 4 files changed, 302 insertions(+), 256 deletions(-)
 delete mode 100644 doc/guides/howto/af_xdp_cni.rst
 create mode 100644 doc/guides/howto/af_xdp_dp.rst

diff --git a/doc/guides/howto/af_xdp_cni.rst b/doc/guides/howto/af_xdp_cni.rst
deleted file mode 100644
index a1a6d5b99c..0000000000
--- a/doc/guides/howto/af_xdp_cni.rst
+++ /dev/null
@@ -1,253 +0,0 @@
-.. SPDX-License-Identifier: BSD-3-Clause
-   Copyright(c) 2023 Intel Corporation.
-
-Using a CNI with the AF_XDP driver
-==================================
-
-Introduction
-------------
-
-CNI, the Container Network Interface, is a technology for configuring
-container network interfaces
-and which can be used to setup Kubernetes networking.
-AF_XDP is a Linux socket Address Family that enables an XDP program
-to redirect packets to a memory buffer in userspace.
-
-This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within
-a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies.
-
-.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
-
-
-Background
-----------
-
-The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program
-onto the kernel netdev to be used by the PMD.
-This operation requires root or escalated Linux privileges
-and thus prevents the PMD from working in an unprivileged container.
-The AF_XDP CNI plugin handles this situation
-by providing a device plugin that performs the program loading.
-
-At a technical level the CNI opens a Unix Domain Socket and listens for a client
-to make requests over that socket.
-A DPDK application acting as a client connects and initiates a configuration "handshake".
-The client then receives a file descriptor which points to the XSKMAP
-associated with the loaded eBPF program.
-The XSKMAP is a BPF map of AF_XDP sockets (XSK).
-The client can then proceed with creating an AF_XDP socket
-and inserting that socket into the XSKMAP pointed to by the descriptor.
-
-The EAL vdev argument ``use_cni`` is used to indicate that the user wishes
-to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor
-from the CNI.
-When this flag is set,
-the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag
-should be used when creating the socket
-to instruct libbpf not to load the default libbpf program on the netdev.
-Instead the loading is handled by the CNI.
-
-.. note::
-
-   The Unix Domain Socket file path appear in the end user is "/tmp/afxdp.sock".
-
-
-Prerequisites
--------------
-
-Docker and container prerequisites:
-
-* Set up the device plugin
-  as described in the instructions for `AF_XDP Plugin for Kubernetes`_.
-
-* The Docker image should contain the libbpf and libxdp libraries,
-  which are dependencies for AF_XDP,
-  and should include support for the ``ethtool`` command.
-
-* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and ``CAP_BPF``
-  for AF_XDP along with support for hugepages.
-
-* Increase locked memory limit so containers have enough memory for packet buffers.
-  For example:
-
-  .. code-block:: console
-
-     cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf
-     [Service]
-     LimitMEMLOCK=infinity
-     EOF
-
-* dpdk-testpmd application should have AF_XDP feature enabled.
-
-  For further information see the docs for the: :doc:`../../nics/af_xdp`.
-
-
-Example
--------
-
-Howto run dpdk-testpmd with CNI plugin:
-
-* Clone the CNI plugin
-
-  .. code-block:: console
-
-     # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git
-
-* Build the CNI plugin
-
-  .. code-block:: console
-
-     # cd afxdp-plugins-for-kubernetes/
-     # make build
-
-  .. note::
-
-     CNI plugin has a dependence on the config.json.
-
-  Sample Config.json
-
-  .. code-block:: json
-
-     {
-        "logLevel":"debug",
-        "logFile":"afxdp-dp-e2e.log",
-        "pools":[
-           {
-              "name":"e2e",
-              "mode":"primary",
-              "timeout":30,
-              "ethtoolCmds" : ["-L -device- combined 1"],
-              "devices":[
-                 {
-                    "name":"ens785f0"
-                 }
-              ]
-           }
-        ]
-     }
-
-  For further reference please use the `config.json`_
-
-  .. _config.json: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/config.json
-
-* Create the Network Attachment definition
-
-  .. code-block:: console
-
-     # kubectl create -f nad.yaml
-
-  Sample nad.yml
-
-  .. code-block:: yaml
-
-      apiVersion: "k8s.cni.cncf.io/v1"
-      kind: NetworkAttachmentDefinition
-      metadata:
-        name: afxdp-e2e-test
-        annotations:
-          k8s.v1.cni.cncf.io/resourceName: afxdp/e2e
-      spec:
-        config: '{
-            "cniVersion": "0.3.0",
-            "type": "afxdp",
-            "mode": "cdq",
-            "logFile": "afxdp-cni-e2e.log",
-            "logLevel": "debug",
-            "ipam": {
-              "type": "host-local",
-              "subnet": "192.168.1.0/24",
-              "rangeStart": "192.168.1.200",
-              "rangeEnd": "192.168.1.216",
-              "routes": [
-                { "dst": "0.0.0.0/0" }
-              ],
-              "gateway": "192.168.1.1"
-            }
-          }'
-
-  For further reference please use the `nad.yaml`_
-
-  .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/nad.yaml
-
-* Build the Docker image
-
-  .. code-block:: console
-
-     # docker build -t afxdp-e2e-test -f Dockerfile .
-
-  Sample Dockerfile:
-
-  .. code-block:: console
-
-     FROM ubuntu:20.04
-     RUN apt-get update -y
-     RUN apt install build-essential libelf-dev -y
-     RUN apt-get install iproute2  acl -y
-     RUN apt install python3-pyelftools ethtool -y
-     RUN apt install libnuma-dev libjansson-dev libpcap-dev net-tools -y
-     RUN apt-get install clang llvm -y
-     COPY ./libbpf<version>.tar.gz /tmp
-     RUN cd /tmp && tar -xvmf libbpf<version>.tar.gz && cd libbpf/src && make install
-     COPY ./libxdp<version>.tar.gz /tmp
-     RUN cd /tmp && tar -xvmf libxdp<version>.tar.gz && cd libxdp && make install
-
-  .. note::
-
-     All the files that need to COPY-ed should be in the same directory as the Dockerfile
-
-* Run the Pod
-
-  .. code-block:: console
-
-     # kubectl create -f pod.yaml
-
-  Sample pod.yaml:
-
-  .. code-block:: yaml
-
-     apiVersion: v1
-     kind: Pod
-     metadata:
-       name: afxdp-e2e-test
-       annotations:
-         k8s.v1.cni.cncf.io/networks: afxdp-e2e-test
-     spec:
-       containers:
-       - name: afxdp
-         image: afxdp-e2e-test:latest
-         imagePullPolicy: Never
-         env:
-         - name: LD_LIBRARY_PATH
-           value: /usr/lib64/:/usr/local/lib/
-         command: ["tail", "-f", "/dev/null"]
-         securityContext:
-          capabilities:
-             add:
-               - CAP_NET_RAW
-               - CAP_BPF
-         resources:
-           requests:
-             hugepages-2Mi: 2Gi
-             memory: 2Gi
-             afxdp/e2e: '1'
-           limits:
-             hugepages-2Mi: 2Gi
-             memory: 2Gi
-             afxdp/e2e: '1'
-
-  For further reference please use the `pod.yaml`_
-
-  .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/pod-1c1d.yaml
-
-* Run DPDK with a command like the following:
-
-  .. code-block:: console
-
-     kubectl exec -i <Pod name> --container <containers name> -- \
-           /<Path>/dpdk-testpmd -l 0,1 --no-pci \
-           --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
-           -- --no-mlockall --in-memory
-
-For further reference please use the `e2e`_ test case in `AF_XDP Plugin for Kubernetes`_
-
-  .. _e2e: https://github.com/intel/afxdp-plugins-for-kubernetes/tree/v0.0.2/test/e2e
diff --git a/doc/guides/howto/af_xdp_dp.rst b/doc/guides/howto/af_xdp_dp.rst
new file mode 100644
index 0000000000..7166d904bd
--- /dev/null
+++ b/doc/guides/howto/af_xdp_dp.rst
@@ -0,0 +1,299 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+   Copyright(c) 2023 Intel Corporation.
+
+Using the AF_XDP driver in Kubernetes
+=====================================
+
+Introduction
+------------
+
+Two infrastructure components are needed in order to provision a pod that is
+using the AF_XDP PMD in Kubernetes:
+
+1. AF_XDP Device Plugin (DP).
+2. AF_XDP Container Network Interface (CNI) binary.
+
+Both of these components are available through the `AF_XDP Device Plugin for Kubernetes`_
+repository.
+
+The AF_XDP DP provisions and advertises networking interfaces to Kubernetes,
+while the CNI configures and plumbs network interfaces for the Pod.
+
+This document explains how to use the `AF_XDP Device Plugin for Kubernetes`_ with
+a DPDK application using the :doc:`../nics/af_xdp`.
+
+.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
+
+Background
+----------
+
+The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program
+onto the kernel netdev to be used by the PMD.
+This operation requires root or escalated Linux privileges
+and thus prevents the PMD from working in an unprivileged container.
+The AF_XDP Device Plugin handles this situation
+by managing the eBPF program(s) on behalf of the Pod, outside of the pod context.
+
+At a technical level the AF_XDP Device Plugin opens a Unix Domain Socket (UDS) and listens for a client
+to make requests over that socket.
+A DPDK application acting as a client connects and initiates a configuration "handshake".
+After some validation on the Device Plugin side, the client receives a file descriptor which points to the XSKMAP
+associated with the loaded eBPF program.
+The XSKMAP is an eBPF map of AF_XDP sockets (XSK).
+The client can then proceed with creating an AF_XDP socket
+and inserting that socket into the XSKMAP pointed to by the descriptor.
+
+The EAL vdev argument ``use_cni`` is used to indicate that the user wishes
+to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor
+from the CNI.
+When this flag is set,
+the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag
+should be used when creating the socket
+to instruct libbpf not to load the default libbpf program on the netdev.
+Instead the loading is handled by the AF_XDP Device Plugin.
+
+Limitations
+-----------
+
+For DPDK versions <= v23.11 the Unix Domain Socket file path appears in
+the pod at "/tmp/afxdp.sock". The handshake implementation in the AF_XDP PMD
+is only compatible with the AF_XDP Device Plugin up to commit id `38317c2`_
+and the pod is limited to a single netdev.
+
+.. note::
+
+    DPDK AF_XDP PMD <= v23.11 will not work with the latest version of the
+    AF_XDP Device Plugin.
+
+The issue is if a single pod requests different devices from different pools it
+results in multiple UDS servers serving the pod with the container using only a
+single mount point for their UDS as ``/tmp/afxdp.sock``. This means that at best one
+device might be able to complete the handshake. This has been fixed in the AF_XDP
+Device Plugin so that the mount point in the pods for the UDS appear at
+``/tmp/afxdp_dp/<netdev>/afxdp.sock``. Later versions of DPDK fix this hardcoded path
+in the PMD alongside the ``use_cni`` parameter.
+
+.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669
+
+
+Prerequisites
+-------------
+
+Device Plugin and DPDK container prerequisites:
+
+* Create a DPDK container image.
+
+* Set up the device plugin and prepare the Pod Spec as described in
+  the instructions for `AF_XDP Device Plugin for Kubernetes`_.
+
+* The Docker image should contain the libbpf and libxdp libraries,
+  which are dependencies for AF_XDP,
+  and should include support for the ``ethtool`` command.
+
+* The Pod should have enabled the capabilities ``CAP_NET_RAW`` for
+  AF_XDP socket creation, ``IPC_LOCK`` for umem creation and
+  ``CAP_BPF`` (for Kernel < 5.19) along with support for hugepages.
+
+  .. note::
+
+    For Kernel versions < 5.19, all BPF sys calls required CAP_BPF, to access maps shared
+    between the eBFP program and the userspace program. Kernels >= 5.19, only requires CAP_BPF
+    for map creation (BPF_MAP_CREATE) and loading programs (BPF_PROG_LOAD).
+
+* Increase locked memory limit so containers have enough memory for packet buffers.
+  For example:
+
+  .. code-block:: console
+
+     cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf
+     [Service]
+     LimitMEMLOCK=infinity
+     EOF
+
+* dpdk-testpmd application should have AF_XDP feature enabled.
+
+  For further information see the docs for the: :doc:`../../nics/af_xdp`.
+
+
+Example
+-------
+
+Build a DPDK container image (using Docker)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Create a Dockerfile (should be placed in top level DPDK directory):
+
+  .. code-block:: console
+
+    FROM fedora:38
+
+    # Setup container to build DPDK applications
+    RUN dnf -y upgrade && dnf -y install \
+        libbsd-devel \
+        numactl-libs \
+        libbpf-devel \
+        libbpf \
+        meson \
+        ninja-build \
+        libxdp-devel \
+        libxdp \
+        numactl-devel \
+        python3-pyelftools \
+        python38 \
+        iproute
+    RUN dnf groupinstall -y 'Development Tools'
+
+    # Create DPDK dir and copy over sources
+    # Create DPDK dir and copy over sources
+    COPY ./ /dpdk
+    WORKDIR /dpdk
+
+    # Build DPDK
+    RUN meson setup build
+    RUN ninja -C build
+
+2. Build a DPDK container image (using Docker)
+
+  .. code-block:: console
+
+    # docker build -t dpdk -f Dockerfile
+
+Run dpdk-testpmd with the AF_XDP Device Plugin + CNI
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Clone the AF_XDP Device plugin and CNI
+
+  .. code-block:: console
+
+     # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git
+
+  .. note::
+
+    Ensure you have the AF_XDP Device Plugin + CNI prerequisites installed.
+
+* Build the AF_XDP Device plugin and CNI
+
+  .. code-block:: console
+
+     # cd afxdp-plugins-for-kubernetes/
+     # make image
+
+* Make sure to modify the image used by the `daemonset.yml`_ file in the deployments directory with
+  the following configuration:
+
+   .. _daemonset.yml : https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/deployments/daemonset.yml
+
+  .. code-block:: yaml
+
+    image: afxdp-device-plugin:latest
+
+  .. note::
+
+    This will select the AF_XDP DP image that was built locally. Detailed configuration
+    options can be found in the AF_XDP Device Plugin `readme`_ .
+
+  .. _readme: https://github.com/intel/afxdp-plugins-for-kubernetes#readme
+
+* Deploy the AF_XDP Device Plugin and CNI
+
+  .. code-block:: console
+
+    # kubectl create -f deployments/daemonset.yml
+
+* Create the Network Attachment definition
+
+  .. code-block:: console
+
+     # kubectl create -f nad.yaml
+
+  Sample nad.yml
+
+  .. code-block:: yaml
+
+    apiVersion: "k8s.cni.cncf.io/v1"
+    kind: NetworkAttachmentDefinition
+    metadata:
+      name: afxdp-network
+      annotations:
+        k8s.v1.cni.cncf.io/resourceName: afxdp/myPool
+    spec:
+      config: '{
+          "cniVersion": "0.3.0",
+          "type": "afxdp",
+          "mode": "primary",
+          "logFile": "afxdp-cni.log",
+          "logLevel": "debug",
+          "ethtoolCmds" : ["-N -device- rx-flow-hash udp4 fn",
+                           "-N -device- flow-type udp4 dst-port 2152 action 22"
+                        ],
+          "ipam": {
+            "type": "host-local",
+            "subnet": "192.168.1.0/24",
+            "rangeStart": "192.168.1.200",
+            "rangeEnd": "192.168.1.220",
+            "routes": [
+              { "dst": "0.0.0.0/0" }
+            ],
+            "gateway": "192.168.1.1"
+          }
+        }'
+
+  For further reference please use the example provided by the AF_XDP DP `nad.yaml`_
+
+  .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/network-attachment-definition.yaml
+
+* Run the Pod
+
+  .. code-block:: console
+
+     # kubectl create -f pod.yaml
+
+  Sample pod.yaml:
+
+  .. code-block:: yaml
+
+    apiVersion: v1
+    kind: Pod
+    metadata:
+     name: dpdk
+     annotations:
+       k8s.v1.cni.cncf.io/networks: afxdp-network
+    spec:
+      containers:
+      - name: testpmd
+        image: dpdk:latest
+        command: ["tail", "-f", "/dev/null"]
+        securityContext:
+          capabilities:
+            add:
+              - NET_RAW
+              - IPC_LOCK
+        resources:
+          requests:
+            afxdp/myPool: '1'
+          limits:
+            hugepages-1Gi: 2Gi
+            cpu: 2
+            memory: 256Mi
+            afxdp/myPool: '1'
+        volumeMounts:
+        - name: hugepages
+          mountPath: /dev/hugepages
+      volumes:
+      - name: hugepages
+        emptyDir:
+          medium: HugePages
+
+  For further reference please use the `pod.yaml`_
+
+  .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/pod-spec.yaml
+
+* Run DPDK with a command like the following:
+
+  .. code-block:: console
+
+     kubectl exec -i <Pod name> --container <containers name> -- \
+           /<Path>/dpdk-testpmd -l 0,1 --no-pci \
+           --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
+           --no-mlockall --in-memory \
+           -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index 71a3381c36..a7692e8a97 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -8,7 +8,7 @@ HowTo Guides
     :maxdepth: 2
     :numbered:
 
-    af_xdp_cni
+    af_xdp_dp
     lm_bond_virtio_sriov
     lm_virtio_vhost_user
     flow_bifurcation
diff --git a/doc/guides/nics/af_xdp.rst b/doc/guides/nics/af_xdp.rst
index 1932525d4d..4dd9c73742 100644
--- a/doc/guides/nics/af_xdp.rst
+++ b/doc/guides/nics/af_xdp.rst
@@ -155,9 +155,9 @@ use_cni
 ~~~~~~~
 
 The EAL vdev argument ``use_cni`` is used to indicate that the user wishes to
-enable the `AF_XDP Plugin for Kubernetes`_ within a DPDK application.
+enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod.
 
-.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
+.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
 
 .. code-block:: console
 
-- 
2.41.0