[dpdk-dev] [PATCH 4/5] doc: changes for new pcapng and dumpcap

[Stephen Hemminger <stephen@networkplumber.org>]

Describe the new packet capture library and utilities

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 doc/api/doxy-api-index.md                     |  1 +
 doc/api/doxy-api.conf.in                      |  1 +
 doc/guides/howto/packet_capture_framework.rst | 67 ++++++++--------
 doc/guides/prog_guide/index.rst               |  1 +
 doc/guides/prog_guide/pcapng_lib.rst          | 24 ++++++
 doc/guides/prog_guide/pdump_lib.rst           | 28 +++++--
 doc/guides/rel_notes/release_21_11.rst        | 10 +++
 doc/guides/tools/dumpcap.rst                  | 80 +++++++++++++++++++
 doc/guides/tools/index.rst                    |  1 +
 9 files changed, 174 insertions(+), 39 deletions(-)
 create mode 100644 doc/guides/prog_guide/pcapng_lib.rst
 create mode 100644 doc/guides/tools/dumpcap.rst

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 1992107a0356..ee07394d1c78 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -223,3 +223,4 @@ The public API headers are grouped by topics:
   [experimental APIs]  (@ref rte_compat.h),
   [ABI versioning]     (@ref rte_function_versioning.h),
   [version]            (@ref rte_version.h)
+  [pcapng]             (@ref rte_pcapng.h)
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 325a0195c6ab..aba17799a9a1 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -58,6 +58,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/metrics \
                           @TOPDIR@/lib/node \
                           @TOPDIR@/lib/net \
+                          @TOPDIR@/lib/pcapng \
                           @TOPDIR@/lib/pci \
                           @TOPDIR@/lib/pdump \
                           @TOPDIR@/lib/pipeline \
diff --git a/doc/guides/howto/packet_capture_framework.rst b/doc/guides/howto/packet_capture_framework.rst
index c31bac52340e..78baa609a021 100644
--- a/doc/guides/howto/packet_capture_framework.rst
+++ b/doc/guides/howto/packet_capture_framework.rst
@@ -1,18 +1,19 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2017 Intel Corporation.
 
-DPDK pdump Library and pdump Tool
-=================================
+DPDK packet capture libraries and tools
+=======================================
 
 This document describes how the Data Plane Development Kit (DPDK) Packet
 Capture Framework is used for capturing packets on DPDK ports. It is intended
 for users of DPDK who want to know more about the Packet Capture feature and
 for those who want to monitor traffic on DPDK-controlled devices.
 
-The DPDK packet capture framework was introduced in DPDK v16.07. The DPDK
-packet capture framework consists of the DPDK pdump library and DPDK pdump
-tool.
-
+The DPDK packet capture framework was introduced in DPDK v16.07 and
+enhanced in 21.1. The DPDK packet capture framework consists of the
+libraries for collecting packets ``librte_pdump`` and writing packets
+to a file ``librte_pcapng``. There are two sample applications:
+``dpdk-dumpcap`` and older ``dpdk-pdump``.
 
 Introduction
 ------------
@@ -22,43 +23,46 @@ allow users to initialize the packet capture framework and to enable or
 disable packet capture. The library works on a multi process communication model and its
 usage is recommended for debugging purposes.
 
-The :ref:`dpdk-pdump <pdump_tool>` tool is developed based on the
-``librte_pdump`` library.  It runs as a DPDK secondary process and is capable
-of enabling or disabling packet capture on DPDK ports. The ``dpdk-pdump`` tool
-provides command-line options with which users can request enabling or
-disabling of the packet capture on DPDK ports.
+The :ref:`librte_pcapng <pcapng_library>` library provides the APIs to format
+packets and write them to a file in Pcapng format.
+
+
+The :ref:`dpdk-dumpcap <dumpcap_tool>` is a tool that captures packets in
+like Wireshark dumpcap does for Linux. It runs as a DPDK secondary process and
+captures packets from one or more interfaces and writes them to a file
+in Pcapng format.  The ``dpdk-dumpcap`` tool is designed to take
+most of the same options as the Wireshark ``dumpcap`` command.
 
-The application which initializes the packet capture framework will be a primary process
-and the application that enables or disables the packet capture will
-be a secondary process. The primary process sends the Rx and Tx packets from the DPDK ports
-to the secondary process.
+Without any options it will use the packet capture framework to
+capture traffic from the first available DPDK port.
 
 In DPDK the ``testpmd`` application can be used to initialize the packet
-capture framework and acts as a server, and the ``dpdk-pdump`` tool acts as a
+capture framework and acts as a server, and the ``dpdk-dumpcap`` tool acts as a
 client. To view Rx or Tx packets of ``testpmd``, the application should be
-launched first, and then the ``dpdk-pdump`` tool. Packets from ``testpmd``
-will be sent to the tool, which then sends them on to the Pcap PMD device and
-that device writes them to the Pcap file or to an external interface depending
-on the command-line option used.
+launched first, and then the ``dpdk-dumpcap`` tool. Packets from ``testpmd``
+will be sent to the tool, and then to the Pcapng file.
 
 Some things to note:
 
-* The ``dpdk-pdump`` tool can only be used in conjunction with a primary
+* All tools using ``librte_pdump`` can only be used in conjunction with a primary
   application which has the packet capture framework initialized already. In
   dpdk, only ``testpmd`` is modified to initialize packet capture framework,
-  other applications remain untouched. So, if the ``dpdk-pdump`` tool has to
+  other applications remain untouched. So, if the ``dpdk-dumpcap`` tool has to
   be used with any application other than the testpmd, the user needs to
   explicitly modify that application to call the packet capture framework
   initialization code. Refer to the ``app/test-pmd/testpmd.c`` code and look
   for ``pdump`` keyword to see how this is done.
 
-* The ``dpdk-pdump`` tool depends on the libpcap based PMD.
+* The ``dpdk-pdump`` tool is an older tool created as demonstration of ``librte_pdump``
+  library. The ``dpdk-pdump`` tool provides more limited functionality and
+  and depends on the Pcap PMD. It is retained only for compatibility reasons;
+  users should use ``dpdk-dumpcap`` instead.
 
 
 Test Environment
 ----------------
 
-The overview of using the Packet Capture Framework and the ``dpdk-pdump`` tool
+The overview of using the Packet Capture Framework and the ``dpdk-dumpcap`` utility
 for packet capturing on the DPDK port in
 :numref:`figure_packet_capture_framework`.
 
@@ -66,13 +70,13 @@ for packet capturing on the DPDK port in
 
 .. figure:: img/packet_capture_framework.*
 
-   Packet capturing on a DPDK port using the dpdk-pdump tool.
+   Packet capturing on a DPDK port using the dpdk-dumpcap utility.
 
 
 Running the Application
 -----------------------
 
-The following steps demonstrate how to run the ``dpdk-pdump`` tool to capture
+The following steps demonstrate how to run the ``dpdk-dumpcap`` tool to capture
 Rx side packets on dpdk_port0 in :numref:`figure_packet_capture_framework` and
 inspect them using ``tcpdump``.
 
@@ -80,16 +84,15 @@ inspect them using ``tcpdump``.
 
      sudo <build_dir>/app/dpdk-testpmd -c 0xf0 -n 4 -- -i --port-topology=chained
 
-#. Launch the pdump tool as follows::
+#. Launch the dpdk-dump as follows::
 
-     sudo <build_dir>/app/dpdk-pdump -- \
-          --pdump 'port=0,queue=*,rx-dev=/tmp/capture.pcap'
+     sudo <build_dir>/app/dpdk-dumpcap -w /tmp/capture.pcapng
 
 #. Send traffic to dpdk_port0 from traffic generator.
-   Inspect packets captured in the file capture.pcap using a tool
-   that can interpret Pcap files, for example tcpdump::
+   Inspect packets captured in the file capture.pcap using a tool such as
+   tcpdump or tshark that can interpret Pcapng files::
 
-     $tcpdump -nr /tmp/capture.pcap
+     $ tcpdump -nr /tmp/capture.pcapng
      reading from file /tmp/capture.pcap, link-type EN10MB (Ethernet)
      11:11:36.891404 IP 4.4.4.4.whois++ > 3.3.3.3.whois++: UDP, length 18
      11:11:36.891442 IP 4.4.4.4.whois++ > 3.3.3.3.whois++: UDP, length 18
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 2dce507f46a3..b440c77c2ba1 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -43,6 +43,7 @@ Programmer's Guide
     ip_fragment_reassembly_lib
     generic_receive_offload_lib
     generic_segmentation_offload_lib
+    pcapng_lib
     pdump_lib
     multi_proc_support
     kernel_nic_interface
diff --git a/doc/guides/prog_guide/pcapng_lib.rst b/doc/guides/prog_guide/pcapng_lib.rst
new file mode 100644
index 000000000000..36379b530a57
--- /dev/null
+++ b/doc/guides/prog_guide/pcapng_lib.rst
@@ -0,0 +1,24 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2016 Intel Corporation.
+
+.. _pcapng_library:
+
+Packet Capture File Writer
+==========================
+
+Pcapng is a library for creating files in Pcapng file format.
+The Pcapng file format is the default capture file format for modern
+network capture processing tools. It can be read by wireshark and tcpdump.
+
+Usage
+-----
+
+Before the library can be used the function ``rte_pcapng_init``
+should be called once to initialize timestamp computation.
+
+
+References
+----------
+* Draft RFC https://www.ietf.org/id/draft-tuexen-opsawg-pcapng-03.html
+
+* Project repository  https://github.com/pcapng/pcapng/
diff --git a/doc/guides/prog_guide/pdump_lib.rst b/doc/guides/prog_guide/pdump_lib.rst
index 62c0b015b2fe..9af91415e5ea 100644
--- a/doc/guides/prog_guide/pdump_lib.rst
+++ b/doc/guides/prog_guide/pdump_lib.rst
@@ -3,10 +3,10 @@
 
 .. _pdump_library:
 
-The librte_pdump Library
-========================
+The Packet Capture Library
+==========================
 
-The ``librte_pdump`` library provides a framework for packet capturing in DPDK.
+The DPDK ``pdump`` library provides a framework for packet capturing in DPDK.
 The library does the complete copy of the Rx and Tx mbufs to a new mempool and
 hence it slows down the performance of the applications, so it is recommended
 to use this library for debugging purposes.
@@ -23,11 +23,19 @@ or disable the packet capture, and to uninitialize it.
 
 * ``rte_pdump_enable()``:
   This API enables the packet capture on a given port and queue.
-  Note: The filter option in the API is a place holder for future enhancements.
+
+* ``rte_pdump_enable_bpf()``
+  This API enables the packet capture on a given port and queue.
+  It also allows setting an optional filter using DPDK BPF interpreter and
+  setting the captured packet length.
 
 * ``rte_pdump_enable_by_deviceid()``:
   This API enables the packet capture on a given device id (``vdev name or pci address``) and queue.
-  Note: The filter option in the API is a place holder for future enhancements.
+
+* ``rte_pdump_enable_bpf_by_deviceid()``
+  This API enables the packet capture on a given device id (``vdev name or pci address``) and queue.
+  It also allows seating an optional filter using DPDK BPF interpreter and
+  setting the captured packet length.
 
 * ``rte_pdump_disable()``:
   This API disables the packet capture on a given port and queue.
@@ -61,6 +69,12 @@ and enables the packet capture by registering the Ethernet RX and TX callbacks f
 and queue combinations. Then the primary process will mirror the packets to the new mempool and enqueue them to
 the rte_ring that secondary process have passed to these APIs.
 
+The packet ring supports one of two formats. The default format enqueues copies of the original packets
+into the rte_ring. If the ``RTE_PDUMP_FLAG_PCAPNG`` is set the mbuf data is extended with header and trailer
+to match the format of Pcapng enhanced packet block. The enhanced packet block has meta-data such as the
+timestamp, port and queue the packet was captured on. It is up to the application consuming the
+packets from the ring to select the format desired.
+
 The library APIs ``rte_pdump_disable()`` and ``rte_pdump_disable_by_deviceid()`` disables the packet capture.
 For the calls to these APIs from secondary process, the library creates the "pdump disable" request and sends
 the request to the primary process over the multi process channel. The primary process takes this request and
@@ -74,5 +88,5 @@ function.
 Use Case: Packet Capturing
 --------------------------
 
-The DPDK ``app/pdump`` tool is developed based on this library to capture packets in DPDK.
-Users can use this as an example to develop their own packet capturing tools.
+The DPDK ``app/dpdk-dumpcap`` utility uses this library
+to capture packets in DPDK.
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 675b5738348b..ee24cbfdb99d 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -62,6 +62,16 @@ New Features
   * Added bus-level parsing of the devargs syntax.
   * Kept compatibility with the legacy syntax as parsing fallback.
 
+* **Enhance Packet capture.**
+
+  * New dpdk-dumpcap program that has most of the features of the
+    wireshark dumpcap utility including capture of multiple interfaces,
+    stopping after number of bytes, packets.
+  * New library for writing pcapng packet capture files.
+  * Enhancement to the pdump library to support:
+    * Packet filter with BPF.
+    * Pcapng format with timestamps and meta-data.
+    * Fixes packet capture with stripped VLAN tags.
 
 Removed Items
 -------------
diff --git a/doc/guides/tools/dumpcap.rst b/doc/guides/tools/dumpcap.rst
new file mode 100644
index 000000000000..90993bd5be5e
--- /dev/null
+++ b/doc/guides/tools/dumpcap.rst
@@ -0,0 +1,80 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Microsoft Corporation.
+
+.. _dumpcap_tool:
+
+dpdk-dumpcap Application
+========================
+
+The ``dpdk-dumpcap`` tool is a Data Plane Development Kit (DPDK)
+network traffic dump tool.  The interface is similar to  the dumpcap tool in Wireshark.
+It runs as a secondary DPDK process and lets you capture packets that are
+coming into and out of a DPDK primary process.
+The ``dpdk-dumpcap`` writes files in Pcapng packet format using
+capture file format is pcapng.
+
+Without any options set it will use DPDK to capture traffic from the first
+available DPDK interface and write the received raw packet data, along
+with timestamps into a pcapng file.
+
+If the ``-w`` option is not specified, ``dpdk-dumpcap`` writes to a newly
+create file with a name chosen based on interface name and timestamp.
+If ``-w`` option is specified, then that file is used.
+
+   .. Note::
+      * The ``dpdk-dumpcap`` tool can only be used in conjunction with a primary
+        application which has the packet capture framework initialized already.
+        In dpdk, only the ``testpmd`` is modified to initialize packet capture
+        framework, other applications remain untouched. So, if the ``dpdk-dumpcap``
+        tool has to be used with any application other than the testpmd, user
+        needs to explicitly modify that application to call packet capture
+        framework initialization code. Refer ``app/test-pmd/testpmd.c``
+        code to see how this is done.
+
+      * The ``dpdk-dumpcap`` tool runs as a DPDK secondary process. It exits when
+        the primary application exits.
+
+
+Running the Application
+-----------------------
+
+To list interfaces available for capture use ``--list-interfaces``.
+
+To capture on multiple interfaces at once, use multiple ``-I`` flags.
+
+Example
+-------
+
+.. code-block:: console
+
+   # ./<build_dir>/app/dpdk-dumpcap --list-interfaces
+   0. 000:00:03.0
+   1. 000:00:03.1
+
+   # ./<build_dir>/app/dpdk-dumpcap -I 0000:00:03.0 -c 6 -w /tmp/sample.pcapng
+   Packets captured: 6
+   Packets received/dropped on interface '0000:00:03.0' 6/0
+
+
+Limitations
+-----------
+The following options of Wireshark ``dumpcap`` are not yet implemented:
+
+   * ``-f <capture filter>`` -- needs translation from classic BPF to eBPF.
+
+   * ``-b|--ring-buffer`` -- more complex file management.
+
+   * ``-C <byte_limit>`` -- doesn't make sense in DPDK model.
+
+   * ``-t`` -- use a thread per interface
+
+   * Timestamp type.
+
+   * Link data types. Only EN10MB (Ethernet) is supported.
+
+   * Wireless related options:  ``-I|--monitor-mode`` and  ``-k <freq>``
+
+
+.. Note::
+   * The options to ``dpdk-dumpcap`` are like the Wireshark dumpcap program and
+     are not the same as ``dpdk-pdump`` and other DPDK applications.
diff --git a/doc/guides/tools/index.rst b/doc/guides/tools/index.rst
index 93dde4148e90..b71c12b8f2dd 100644
--- a/doc/guides/tools/index.rst
+++ b/doc/guides/tools/index.rst
@@ -8,6 +8,7 @@ DPDK Tools User Guides
     :maxdepth: 2
     :numbered:
 
+    dumpcap
     proc_info
     pdump
     pmdinfo
-- 
2.30.2