From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 21064A0C43; Wed, 20 Oct 2021 23:44:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 18DB241194; Wed, 20 Oct 2021 23:42:59 +0200 (CEST) Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by mails.dpdk.org (Postfix) with ESMTP id 77A9C41184 for ; Wed, 20 Oct 2021 23:42:55 +0200 (CEST) Received: by mail-pf1-f182.google.com with SMTP id d5so4106342pfu.1 for ; Wed, 20 Oct 2021 14:42:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/frg0txPU6sKcpsJnayzg7ZDbz0HeXx5xBQ1bUTTfdQ=; b=xv61DQI+P8tutHoILT4hN4TvJXeS0RslFKPq/uz1/gxqPNo0Myz4qAeXZipWDdYxMl IlTbSoW7JgRk5WluvsCSvXXgmKSM1qKVT1G+o6bhMtcRI226q5F5bkvBGy/N80rtjlul b4jibuVAux7mC2w2BT4ixtmwS+ygcfaxc0aez3Ntj5baxEw0LuEKXfUKJEfI32ECjVBR hblCNn6ZjhSPmbLYTg0yd/37KTFWBlpxxMdVzKoG2xzbIqMfaXx/rR5cilLN6/Cc2RKf Ajudg2fDo5Ig7mqy16gvWLOiwBENLq4az4D0SHOuKTrWX2Oh+P8M2Raw7OKPSkgxOhnV t6pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/frg0txPU6sKcpsJnayzg7ZDbz0HeXx5xBQ1bUTTfdQ=; b=x5RVRAvyOtclckOehNpqmLCKnalZ8dIWwSon+Yw+lVRH4iiIfFx6QjhZw9KEkA+am4 MbMc3QT/e/w3e4BL2k22b/Rzx6fYk+4zMmztK408J6jtVlPfk/mHJX28cZecapPZI8I6 SutnfG58AgLFVaEQuuoijIcemVomgDdP9WN32mpwA+OMuNkZnQl6rAC6c/0H06L613U1 HlFKxjJWqgc0OkCTp+D1TNmVfaEr/mV5E6TD/5oHervWxwvruKZX5BwsCwZ5hwroh5wA sQ00aSGoPUiMdsq8LvacwUGFDAqsOC9z+cx0Vp3Qke5IgZbN403d+kRRsRYaHdevloyh X7nQ== X-Gm-Message-State: AOAM532G47nVjL0zes1eLaVbvO4ZReOcHNaHufyjKJPVMiEaj/6reU9l q2MBHzf09oDinT6my1lLwdnsstmNEo8O9A== X-Google-Smtp-Source: ABdhPJyTb8sM8tQz2T26Q2jfQjBSrsHdTm3C0vI9ypYKhtBz9xytnov9j92NY5AxiGC91VngHEj+eQ== X-Received: by 2002:a62:9215:0:b0:44c:4de1:f777 with SMTP id o21-20020a629215000000b0044c4de1f777mr1284329pfd.31.1634766173568; Wed, 20 Oct 2021 14:42:53 -0700 (PDT) Received: from hermes.local (204-195-33-123.wavecable.com. [204.195.33.123]) by smtp.gmail.com with ESMTPSA id d20sm3304480pfl.82.2021.10.20.14.42.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Oct 2021 14:42:53 -0700 (PDT) From: Stephen Hemminger To: dev@dpdk.org Cc: Stephen Hemminger , Reshma Pattan Date: Wed, 20 Oct 2021 14:42:35 -0700 Message-Id: <20211020214236.71444-12-stephen@networkplumber.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211020214236.71444-1-stephen@networkplumber.org> References: <20210903004732.109023-1-stephen@networkplumber.org> <20211020214236.71444-1-stephen@networkplumber.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v15 11/12] doc: changes for new pcapng and dumpcap utility X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Describe the new packet capture library and utility. Fix the title line on the pdump documentation. Signed-off-by: Stephen Hemminger Acked-by: Reshma Pattan --- doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + .../howto/img/packet_capture_framework.svg | 96 +++++++++---------- doc/guides/howto/packet_capture_framework.rst | 69 ++++++------- doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/pcapng_lib.rst | 46 +++++++++ doc/guides/prog_guide/pdump_lib.rst | 28 ++++-- doc/guides/rel_notes/release_21_11.rst | 10 ++ doc/guides/tools/dumpcap.rst | 86 +++++++++++++++++ doc/guides/tools/index.rst | 1 + 10 files changed, 251 insertions(+), 88 deletions(-) create mode 100644 doc/guides/prog_guide/pcapng_lib.rst create mode 100644 doc/guides/tools/dumpcap.rst diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md index 29390504318b..a447c1ab4ac0 100644 --- a/doc/api/doxy-api-index.md +++ b/doc/api/doxy-api-index.md @@ -224,3 +224,4 @@ The public API headers are grouped by topics: [experimental APIs] (@ref rte_compat.h), [ABI versioning] (@ref rte_function_versioning.h), [version] (@ref rte_version.h) + [pcapng] (@ref rte_pcapng.h) diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in index 109ec1f6826b..096ebbaf0d1b 100644 --- a/doc/api/doxy-api.conf.in +++ b/doc/api/doxy-api.conf.in @@ -59,6 +59,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/metrics \ @TOPDIR@/lib/node \ @TOPDIR@/lib/net \ + @TOPDIR@/lib/pcapng \ @TOPDIR@/lib/pci \ @TOPDIR@/lib/pdump \ @TOPDIR@/lib/pipeline \ diff --git a/doc/guides/howto/img/packet_capture_framework.svg b/doc/guides/howto/img/packet_capture_framework.svg index a76baf71fdee..1c2646a81096 100644 --- a/doc/guides/howto/img/packet_capture_framework.svg +++ b/doc/guides/howto/img/packet_capture_framework.svg @@ -1,6 +1,4 @@ - - + inkscape:version="1.0.2 (e86c870879, 2021-01-15)" + sodipodi:docname="packet_capture_framework.svg"> + gradientTransform="matrix(1.1457977,0,0,0.99944907,-151.97019,745.05014)" /> + inkscape:window-width="2560" + inkscape:window-height="1414" + inkscape:window-x="0" + inkscape:window-y="0" + inkscape:window-maximized="1" + inkscape:document-rotation="0" /> @@ -296,7 +295,7 @@ image/svg+xml - + @@ -321,15 +320,15 @@ y="790.82452" /> DPDK Primary Application + y="807.3205" + style="font-size:12.5px;line-height:1.25">DPDK Primary Application dpdk-pdumpdpdk-dumpcaptool + id="tspan4193" + style="font-size:12.5px;line-height:1.25">tool PCAP PMD + id="tspan4193-3" + style="font-size:12.5px;line-height:1.25">librte_pcapng dpdk_port0 + id="tspan4193-6" + style="font-size:12.5px;line-height:1.25">dpdk_port0 librte_pdump + id="tspan4193-2" + style="font-size:12.5px;line-height:1.25">librte_pdump + width="108.21974" + height="35.335861" + x="297.9809" + y="985.62219" /> capture.pcap + id="tspan4193-3-2" + style="font-size:12.5px;line-height:1.25">capture.pcapng Traffic Generator + id="tspan4193-3-2-7" + style="font-size:12.5px;line-height:1.25">Traffic Generator ` tool is developed based on the -``librte_pdump`` library. It runs as a DPDK secondary process and is capable -of enabling or disabling packet capture on DPDK ports. The ``dpdk-pdump`` tool -provides command-line options with which users can request enabling or -disabling of the packet capture on DPDK ports. +The :ref:`librte_pcapng ` library provides the APIs to format +packets and write them to a file in Pcapng format. + + +The :ref:`dpdk-dumpcap ` is a tool that captures packets in +like Wireshark dumpcap does for Linux. It runs as a DPDK secondary process and +captures packets from one or more interfaces and writes them to a file +in Pcapng format. The ``dpdk-dumpcap`` tool is designed to take +most of the same options as the Wireshark ``dumpcap`` command. -The application which initializes the packet capture framework will be a primary process -and the application that enables or disables the packet capture will -be a secondary process. The primary process sends the Rx and Tx packets from the DPDK ports -to the secondary process. +Without any options it will use the packet capture framework to +capture traffic from the first available DPDK port. In DPDK the ``testpmd`` application can be used to initialize the packet -capture framework and acts as a server, and the ``dpdk-pdump`` tool acts as a +capture framework and acts as a server, and the ``dpdk-dumpcap`` tool acts as a client. To view Rx or Tx packets of ``testpmd``, the application should be -launched first, and then the ``dpdk-pdump`` tool. Packets from ``testpmd`` -will be sent to the tool, which then sends them on to the Pcap PMD device and -that device writes them to the Pcap file or to an external interface depending -on the command-line option used. +launched first, and then the ``dpdk-dumpcap`` tool. Packets from ``testpmd`` +will be sent to the tool, and then to the Pcapng file. Some things to note: -* The ``dpdk-pdump`` tool can only be used in conjunction with a primary +* All tools using ``librte_pdump`` can only be used in conjunction with a primary application which has the packet capture framework initialized already. In dpdk, only ``testpmd`` is modified to initialize packet capture framework, - other applications remain untouched. So, if the ``dpdk-pdump`` tool has to + other applications remain untouched. So, if the ``dpdk-dumpcap`` tool has to be used with any application other than the testpmd, the user needs to explicitly modify that application to call the packet capture framework initialization code. Refer to the ``app/test-pmd/testpmd.c`` code and look for ``pdump`` keyword to see how this is done. -* The ``dpdk-pdump`` tool depends on the libpcap based PMD. +* The ``dpdk-pdump`` tool is an older tool created as demonstration of ``librte_pdump`` + library. The ``dpdk-pdump`` tool provides more limited functionality and + and depends on the Pcap PMD. It is retained only for compatibility reasons; + users should use ``dpdk-dumpcap`` instead. Test Environment ---------------- -The overview of using the Packet Capture Framework and the ``dpdk-pdump`` tool +The overview of using the Packet Capture Framework and the ``dpdk-dumpcap`` utility for packet capturing on the DPDK port in :numref:`figure_packet_capture_framework`. @@ -66,13 +70,13 @@ for packet capturing on the DPDK port in .. figure:: img/packet_capture_framework.* - Packet capturing on a DPDK port using the dpdk-pdump tool. + Packet capturing on a DPDK port using the dpdk-dumpcap utility. Running the Application ----------------------- -The following steps demonstrate how to run the ``dpdk-pdump`` tool to capture +The following steps demonstrate how to run the ``dpdk-dumpcap`` tool to capture Rx side packets on dpdk_port0 in :numref:`figure_packet_capture_framework` and inspect them using ``tcpdump``. @@ -80,16 +84,15 @@ inspect them using ``tcpdump``. sudo /app/dpdk-testpmd -c 0xf0 -n 4 -- -i --port-topology=chained -#. Launch the pdump tool as follows:: +#. Launch the dpdk-dumpcap as follows:: - sudo /app/dpdk-pdump -- \ - --pdump 'port=0,queue=*,rx-dev=/tmp/capture.pcap' + sudo /app/dpdk-dumpcap -w /tmp/capture.pcapng #. Send traffic to dpdk_port0 from traffic generator. - Inspect packets captured in the file capture.pcap using a tool - that can interpret Pcap files, for example tcpdump:: + Inspect packets captured in the file capture.pcapng using a tool such as + tcpdump or tshark that can interpret Pcapng files:: - $tcpdump -nr /tmp/capture.pcap + $ tcpdump -nr /tmp/capture.pcapng reading from file /tmp/capture.pcap, link-type EN10MB (Ethernet) 11:11:36.891404 IP 4.4.4.4.whois++ > 3.3.3.3.whois++: UDP, length 18 11:11:36.891442 IP 4.4.4.4.whois++ > 3.3.3.3.whois++: UDP, length 18 diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst index 89af28dacb72..a8e8e759ecf2 100644 --- a/doc/guides/prog_guide/index.rst +++ b/doc/guides/prog_guide/index.rst @@ -44,6 +44,7 @@ Programmer's Guide ip_fragment_reassembly_lib generic_receive_offload_lib generic_segmentation_offload_lib + pcapng_lib pdump_lib multi_proc_support kernel_nic_interface diff --git a/doc/guides/prog_guide/pcapng_lib.rst b/doc/guides/prog_guide/pcapng_lib.rst new file mode 100644 index 000000000000..fa1994c96f4d --- /dev/null +++ b/doc/guides/prog_guide/pcapng_lib.rst @@ -0,0 +1,46 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Microsoft Corporation + +.. _pcapng_library: + +Packet Capture Next Generation Library +====================================== + +Exchanging packet traces becomes more and more critical every day. +The de facto standard for this is the format define by libpcap; +but that format is rather old and is lacking in functionality +for more modern applications. The `Pcapng file format`_ +is the default capture file format for modern network capture +processing tools such as `wireshark`_ (can also be read by `tcpdump`_). + +The Pcapng library is a an API for formatting packet data into +into a Pcapng file. +The format conforms to the current `Pcapng RFC`_ standard. +It is designed to be integrated with the packet capture library. + +Usage +----- + +Before the library can be used the function ``rte_pcapng_init`` +should be called once to initialize timestamp computation. + +The output stream is created with ``rte_pcapng_fdopen``, +and should be closed with ``rte_pcapng_close``. + +The library requires a DPDK mempool to allocate mbufs. The mbufs +need to be able to accommodate additional space for the pcapng packet +format header and trailer information; the function ``rte_pcapng_mbuf_size`` +should be used to determine the lower bound based on MTU. + +Collecting packets is done in two parts. The function ``rte_pcapng_copy`` +is used to format and copy mbuf data and ``rte_pcapng_write_packets`` +writes a burst of packets to the output file. + +The function ``rte_pcapng_write_stats`` can be used to write +statistics information into the output file. The summary statistics +information is automatically added by ``rte_pcapng_close``. + +.. _Tcpdump: https://tcpdump.org/ +.. _Wireshark: https://wireshark.org/ +.. _Pcapng file format: https://github.com/pcapng/pcapng/ +.. _Pcapng RFC: https://datatracker.ietf.org/doc/html/draft-tuexen-opsawg-pcapng diff --git a/doc/guides/prog_guide/pdump_lib.rst b/doc/guides/prog_guide/pdump_lib.rst index 62c0b015b2fe..f3ff8fd828dc 100644 --- a/doc/guides/prog_guide/pdump_lib.rst +++ b/doc/guides/prog_guide/pdump_lib.rst @@ -3,10 +3,10 @@ .. _pdump_library: -The librte_pdump Library -======================== +Packet Capture Library +====================== -The ``librte_pdump`` library provides a framework for packet capturing in DPDK. +The DPDK ``pdump`` library provides a framework for packet capturing in DPDK. The library does the complete copy of the Rx and Tx mbufs to a new mempool and hence it slows down the performance of the applications, so it is recommended to use this library for debugging purposes. @@ -23,11 +23,19 @@ or disable the packet capture, and to uninitialize it. * ``rte_pdump_enable()``: This API enables the packet capture on a given port and queue. - Note: The filter option in the API is a place holder for future enhancements. + +* ``rte_pdump_enable_bpf()`` + This API enables the packet capture on a given port and queue. + It also allows setting an optional filter using DPDK BPF interpreter and + setting the captured packet length. * ``rte_pdump_enable_by_deviceid()``: This API enables the packet capture on a given device id (``vdev name or pci address``) and queue. - Note: The filter option in the API is a place holder for future enhancements. + +* ``rte_pdump_enable_bpf_by_deviceid()`` + This API enables the packet capture on a given device id (``vdev name or pci address``) and queue. + It also allows setting an optional filter using DPDK BPF interpreter and + setting the captured packet length. * ``rte_pdump_disable()``: This API disables the packet capture on a given port and queue. @@ -61,6 +69,12 @@ and enables the packet capture by registering the Ethernet RX and TX callbacks f and queue combinations. Then the primary process will mirror the packets to the new mempool and enqueue them to the rte_ring that secondary process have passed to these APIs. +The packet ring supports one of two formats. The default format enqueues copies of the original packets +into the rte_ring. If the ``RTE_PDUMP_FLAG_PCAPNG`` is set the mbuf data is extended with header and trailer +to match the format of Pcapng enhanced packet block. The enhanced packet block has meta-data such as the +timestamp, port and queue the packet was captured on. It is up to the application consuming the +packets from the ring to select the format desired. + The library APIs ``rte_pdump_disable()`` and ``rte_pdump_disable_by_deviceid()`` disables the packet capture. For the calls to these APIs from secondary process, the library creates the "pdump disable" request and sends the request to the primary process over the multi process channel. The primary process takes this request and @@ -74,5 +88,5 @@ function. Use Case: Packet Capturing -------------------------- -The DPDK ``app/pdump`` tool is developed based on this library to capture packets in DPDK. -Users can use this as an example to develop their own packet capturing tools. +The DPDK ``app/dpdk-dumpcap`` utility uses this library +to capture packets in DPDK. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 30175246c74a..c91f36500a7c 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -189,6 +189,16 @@ New Features * Added tests to verify tunnel header verification in IPsec inbound. * Added tests to verify inner checksum. +* **Revised packet capture framework.** + + * New dpdk-dumpcap program that has most of the features of the + wireshark dumpcap utility including: capture of multiple interfaces, + filtering, and stopping after number of bytes, packets. + * New library for writing pcapng packet capture files. + * Enhancements to the pdump library to support: + * Packet filter with BPF. + * Pcapng format with timestamps and meta-data. + * Fixes packet capture with stripped VLAN tags. Removed Items ------------- diff --git a/doc/guides/tools/dumpcap.rst b/doc/guides/tools/dumpcap.rst new file mode 100644 index 000000000000..664ea0c79802 --- /dev/null +++ b/doc/guides/tools/dumpcap.rst @@ -0,0 +1,86 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2020 Microsoft Corporation. + +.. _dumpcap_tool: + +dpdk-dumpcap Application +======================== + +The ``dpdk-dumpcap`` tool is a Data Plane Development Kit (DPDK) +network traffic dump tool. The interface is similar to the dumpcap tool in Wireshark. +It runs as a secondary DPDK process and lets you capture packets that are +coming into and out of a DPDK primary process. +The ``dpdk-dumpcap`` writes files in Pcapng packet format using +capture file format is pcapng. + +Without any options set it will use DPDK to capture traffic from the first +available DPDK interface and write the received raw packet data, along +with timestamps into a pcapng file. + +If the ``-w`` option is not specified, ``dpdk-dumpcap`` writes to a newly +create file with a name chosen based on interface name and timestamp. +If ``-w`` option is specified, then that file is used. + + .. Note:: + * The ``dpdk-dumpcap`` tool can only be used in conjunction with a primary + application which has the packet capture framework initialized already. + In dpdk, only the ``testpmd`` is modified to initialize packet capture + framework, other applications remain untouched. So, if the ``dpdk-dumpcap`` + tool has to be used with any application other than the testpmd, user + needs to explicitly modify that application to call packet capture + framework initialization code. Refer ``app/test-pmd/testpmd.c`` + code to see how this is done. + + * The ``dpdk-dumpcap`` tool runs as a DPDK secondary process. It exits when + the primary application exits. + + +Running the Application +----------------------- + +To list interfaces available for capture use ``--list-interfaces``. + +To filter packets in style of *tshark* use the ``-f`` flag. + +To capture on multiple interfaces at once, use multiple ``-I`` flags. + +Example +------- + +.. code-block:: console + + # .//app/dpdk-dumpcap --list-interfaces + 0. 000:00:03.0 + 1. 000:00:03.1 + + # .//app/dpdk-dumpcap -I 0000:00:03.0 -c 6 -w /tmp/sample.pcapng + Packets captured: 6 + Packets received/dropped on interface '0000:00:03.0' 6/0 + + # .//app/dpdk-dumpcap -f 'tcp port 80' + Packets captured: 6 + Packets received/dropped on interface '0000:00:03.0' 10/8 + + +Limitations +----------- +The following option of Wireshark ``dumpcap`` is not yet implemented: + + * ``-b|--ring-buffer`` -- more complex file management. + +The following options do not make sense in the context of DPDK. + + * ``-C `` -- its a kernel thing + + * ``-t`` -- use a thread per interface + + * Timestamp type. + + * Link data types. Only EN10MB (Ethernet) is supported. + + * Wireless related options: ``-I|--monitor-mode`` and ``-k `` + + +.. Note:: + * The options to ``dpdk-dumpcap`` are like the Wireshark dumpcap program and + are not the same as ``dpdk-pdump`` and other DPDK applications. diff --git a/doc/guides/tools/index.rst b/doc/guides/tools/index.rst index 93dde4148e90..b71c12b8f2dd 100644 --- a/doc/guides/tools/index.rst +++ b/doc/guides/tools/index.rst @@ -8,6 +8,7 @@ DPDK Tools User Guides :maxdepth: 2 :numbered: + dumpcap proc_info pdump pmdinfo -- 2.30.2