Re: [dpdk-dev] [PATCH v5 10/10] doc: add application usage guide for l2fwd-event

[Jerin Jacob <jerinjacobk@gmail.com>]

On Thu, Oct 3, 2019 at 2:29 AM <pbhagavatula@marvell.com> wrote:
>
> From: Sunil Kumar Kori <skori@marvell.com>
>
> Add documentation for l2fwd-event example.
> Update release notes.
>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>

# Please fix the typos through with the following command

aspell --lang=en_US --check doc/guides/sample_app_ug/l2_forward_event.rst

# Remove the following warning.

$ make  doc-guides-html
doc/guides/sample_app_ug/l2_forward_real_virtual.rst:39: WARNING:
duplicate label figure_l2_fwd_benchmark_setup,


> ---
>  MAINTAINERS                                   |   1 +
>  doc/guides/rel_notes/release_19_11.rst        |   6 +
>  doc/guides/sample_app_ug/index.rst            |   1 +
>  doc/guides/sample_app_ug/intro.rst            |   5 +
>  doc/guides/sample_app_ug/l2_forward_event.rst | 755 ++++++++++++++++++
>  5 files changed, 768 insertions(+)
>  create mode 100644 doc/guides/sample_app_ug/l2_forward_event.rst
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 292ac10c3..94a49b812 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1461,6 +1461,7 @@ F: examples/l2fwd-cat/
>  M: Sunil Kumar Kori <skori@marvell.com>
>  M: Pavan Nikhilesh <pbhagavatula@marvell.com>
>  F: examples/l2fwd-event/
> +F: doc/guides/sample_app_ug/l2_forward_event.rst
>  T: git://dpdk.org/next/dpdk-next-eventdev
>
>  F: examples/l3fwd/
> diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> index 27cfbd9e3..071593e4d 100644
> --- a/doc/guides/rel_notes/release_19_11.rst
> +++ b/doc/guides/rel_notes/release_19_11.rst
> @@ -56,6 +56,12 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =========================================================
>
> +* **Added new example l2fwd-event**

Added new example l2fwd-event application

> +
> +  Added an example application `l2fwd-event` that adds event device support to
> +  traditional  l2fwd example. The default poll mode is also preserved for
> +  readability.

Please change the last sentence to the following or something similar

"It demonstrates usage of poll and event mode IO mechanism under a
single application"



> +
>
>  Removed Items
>  -------------
> diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
> index f23f8f59e..41388231a 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -26,6 +26,7 @@ Sample Applications User Guides
>      l2_forward_crypto
>      l2_forward_job_stats
>      l2_forward_real_virtual
> +    l2_forward_event
>      l2_forward_cat
>      l3_forward
>      l3_forward_power_man
> diff --git a/doc/guides/sample_app_ug/intro.rst b/doc/guides/sample_app_ug/intro.rst
> index 90704194a..84591c0a1 100644
> --- a/doc/guides/sample_app_ug/intro.rst
> +++ b/doc/guides/sample_app_ug/intro.rst
> @@ -87,6 +87,11 @@ examples are highlighted below.
>    forwarding, or ``l2fwd`` application does forwarding based on Ethernet MAC
>    addresses like a simple switch.
>
> +* :doc:`Network Layer 2 forwarding<l2_forward_event>`: The Network Layer 2
> +  forwarding, or ``l2fwd-event`` application does forwarding based on Ethernet MAC
> +  addresses like a simple switch. It demonstrate usage of poll and event mode Rx/Tx
> +  mechanism.


Please change the last sentence to the following or something similar

"It demonstrates usage of poll and event mode IO mechanism under a
single application"

> +
>  * :doc:`Network Layer 3 forwarding<l3_forward>`: The Network Layer3
>    forwarding, or ``l3fwd`` application does forwarding based on Internet
>    Protocol, IPv4 or IPv6 like a simple router.
> diff --git a/doc/guides/sample_app_ug/l2_forward_event.rst b/doc/guides/sample_app_ug/l2_forward_event.rst
> new file mode 100644
> index 000000000..250d16887
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/l2_forward_event.rst
> @@ -0,0 +1,755 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2010-2014 Intel Corporation.
> +
> +.. _l2_fwd_event_app:
> +
> +L2 Forwarding Eventdev Sample Application
> +=========================================
> +
> +The L2 Forwarding eventdev sample application is a simple example of packet
> +processing using the Data Plane Development Kit (DPDK) to demonstrate usage of
> +poll and event mode packet I/O mechanism.
> +
> +Overview
> +--------
> +
> +The L2 Forwarding eventdev sample application, performs L2 forwarding for each
> +packet that is received on an RX_PORT. The destination port is the adjacent port
> +from the enabled portmask, that is, if the first four ports are enabled (portmask=0x0f),
> +ports 1 and 2 forward into each other, and ports 3 and 4 forward into each other.
> +Also, if MAC addresses updating is enabled, the MAC addresses are affected as follows:
> +
> +*   The source MAC address is replaced by the TX_PORT MAC address
> +
> +*   The destination MAC address is replaced by  02:00:00:00:00:TX_PORT_ID
> +
> +Appliation receives packets from RX_PORT using below mentioned methods:
> +
> +*   Poll mode
> +
> +*   Eventdev mode (default)
> +
> +This application can be used to benchmark performance using a traffic-generator,
> +as shown in the :numref:`figure_l2_fwd_benchmark_setup`.
> +
> +.. _figure_l2_fwd_benchmark_setup:
> +
> +.. figure:: img/l2_fwd_benchmark_setup.*

Looks like there is a typo in the original l2fwd application image.
The diagram shows NUT instead of DUT. Please send a separate patch.


> +
> +   Performance Benchmark Setup (Basic Environment)
> +
> +Compiling the Application
> +-------------------------
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``l2fwd-event`` sub-directory.
> +
> +Running the Application
> +-----------------------
> +
> +The application requires a number of command line options:
> +
> +.. code-block:: console
> +
> +    ./build/l2fwd-event [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating --mode=MODE --eventq-sync=SYNC_MODE
> +
> +where,
> +
> +*   p PORTMASK: A hexadecimal bitmask of the ports to configure
> +
> +*   q NQ: A number of queues (=ports) per lcore (default is 1)
> +
> +*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
> +
> +*   --mode=MODE: Packet transfer mode for I/O, poll or eventdev. Eventdev by default.
> +
> +*   --eventq-sync=SYNC_MODE: Event queue synchronization method, Ordered or Atomic. Atomic by default.
> +
> +Sample usage commands are given below to run the application into different mode:
> +
> +Poll mode on linux environment with 4 lcores, 16 ports and 8 RX queues per lcore

Nothing specific to Linux here, so please remove Linux environment


> +Driver Initialization
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +The main part of the code in the main() function relates to the initialization
> +of the driver. To fully understand this code, it is recommended to study the
> +chapters that related to the Poll Mode and Event mode Driver in the
> +*DPDK Programmer's Guide* - Rel 1.4 EAR and the *DPDK API Reference*.
> +
> +.. code-block:: c
> +
> +    if (rte_pci_probe() < 0)
> +        rte_exit(EXIT_FAILURE, "Cannot probe PCI\n");
> +
> +    /* reset l2fwd_dst_ports */
> +
> +    for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++)
> +        l2fwd_dst_ports[portid] = 0;
> +
> +    last_port = 0;
> +
> +    /*
> +     * Each logical core is assigned a dedicated TX queue on each port.
> +     */
> +
> +    RTE_ETH_FOREACH_DEV(portid) {
> +        /* skip ports that are not enabled */
> +
> +        if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
> +           continue;
> +
> +        if (nb_ports_in_mask % 2) {
> +            l2fwd_dst_ports[portid] = last_port;
> +            l2fwd_dst_ports[last_port] = portid;
> +        }
> +        else
> +           last_port = portid;
> +
> +        nb_ports_in_mask++;
> +
> +        rte_eth_dev_info_get((uint8_t) portid, &dev_info);
> +    }
> +
> +Observe that:
> +
> +*   rte_igb_pmd_init_all() simultaneously registers the driver as a PCI driver

Looks like rte_igb_pmd_init_all() from legacy l2fwd doc, Please fix in
both docs.
The original l2fwd doc fix send it as a separate patch.




> +.. code-block:: c
> +
> +        /* Start event device service */
> +        ret = rte_event_dev_service_id_get(eventdev_rsrc.event_d_id,
> +                                           &service_id);
> +        if (ret != -ESRCH && ret != 0)
> +                rte_exit(EXIT_FAILURE, "Error in starting eventdev");
> +
> +        rte_service_runstate_set(service_id, 1);
> +        rte_service_set_runstate_mapped_check(service_id, 0);
> +        eventdev_rsrc.service_id = service_id;


eventdev_rsrc removed recently in code review, Please sync code with
sync in all code sections.


> +.. note::
> +
> +    In the following code, one line for getting the output port requires some
> +    explanation.

Do we need this note: ?

> +
> +During the initialization process, a static array of destination ports
> +(l2fwd_dst_ports[]) is filled such that for each source port, a destination port
> +is assigned that is either the next or previous enabled port from the portmask.
> +If number of ports are odd in portmask then packet from last port will be
> +forwarded to first port i.e. if portmask=0x07, then forwarding will take place
> +like p0--->p1, p1--->p2, p2--->p0.
> +
> +Also to optimize enqueue opeartion, l2fwd_simple_forward() stores incoming mbus
> +upto MAX_PKT_BURST. Once it reaches upto limit, all packets are transmitted to
> +destination ports.
> +
> +.. code-block:: c
> +
> +   static void
> +   l2fwd_simple_forward(struct rte_mbuf *m, uint32_t portid)
> +   {
> +       uint32_t dst_port;
> +       int32_t sent;
> +       struct rte_eth_dev_tx_buffer *buffer;
> +
> +       dst_port = l2fwd_dst_ports[portid];
> +
> +       if (mac_updating)
> +           l2fwd_mac_updating(m, dst_port);
> +
> +       buffer = tx_buffer[dst_port];
> +       sent = rte_eth_tx_buffer(dst_port, 0, buffer, m);
> +       if (sent)
> +       port_statistics[dst_port].tx += sent;
> +   }
> +
> +For this test application, the processing is exactly the same for all packets
> +arriving on the same RX port. Therefore, it would have been possible to call
> +the rte_eth_tx_buffer() function directly from the main loop to send all the
> +received packets on the same TX port, using the burst-oriented send function,
> +which is more efficient.
> +
> +However, in real-life applications (such as, L3 routing),
> +packet N is not necessarily forwarded on the same port as packet N-1.
> +The application is implemented to illustrate that, so the same approach can be
> +reused in a more complex application.
> +
> +To ensure that no packets remain in the tables, each lcore does a draining of TX
> +queue in its main loop. This technique introduces some latency when there are
> +not many packets to send, however it improves performance:
> +
> +.. code-block:: c
> +
> +        cur_tsc = rte_rdtsc();
> +
> +        /*
> +        * TX burst queue drain
> +        */
> +        diff_tsc = cur_tsc - prev_tsc;
> +        if (unlikely(diff_tsc > drain_tsc)) {
> +                for (i = 0; i < qconf->n_rx_port; i++) {
> +                        portid = l2fwd_dst_ports[qconf->rx_port_list[i]];
> +                        buffer = tx_buffer[portid];
> +                        sent = rte_eth_tx_buffer_flush(portid, 0,
> +                                                       buffer);
> +                        if (sent)
> +                                port_statistics[portid].tx += sent;
> +                }
> +
> +                /* if timer is enabled */
> +                if (timer_period > 0) {
> +                        /* advance the timer */
> +                        timer_tsc += diff_tsc;
> +
> +                        /* if timer has reached its timeout */
> +                        if (unlikely(timer_tsc >= timer_period)) {
> +                                /* do this only on master core */
> +                                if (lcore_id == rte_get_master_lcore()) {
> +                                        print_stats();
> +                                        /* reset the timer */
> +                                        timer_tsc = 0;
> +                                }
> +                        }
> +                }
> +
> +                prev_tsc = cur_tsc;
> +        }
> +
> +In the **l2fwd_main_loop_eventdev()** function, the main task is to read ingress
> +packets from the event ports. This is done using the following code:
> +
> +.. code-block:: c
> +
> +        /* Read packet from eventdev */
> +        nb_rx = rte_event_dequeue_burst(event_d_id, event_p_id,
> +                                        events, deq_len, 0);
> +        if (nb_rx == 0) {
> +                rte_pause();
> +                continue;
> +        }
> +
> +        for (i = 0; i < nb_rx; i++) {
> +                mbuf[i] = events[i].mbuf;
> +                rte_prefetch0(rte_pktmbuf_mtod(mbuf[i], void *));
> +        }
> +
> +
> +Before reading packets, deq_len is fetched to ensure correct allowed deq length
> +by the eventdev.
> +The rte_event_dequeue_burst() function writes the mbuf pointers in a local table
> +and returns the number of available mbufs in the table.
> +
> +Then, each mbuf in the table is processed by the l2fwd_eventdev_forward()
> +function. The processing is very simple: process the TX port from the RX port,
> +then replace the source and destination MAC addresses if MAC addresses updating
> +is enabled.
> +
> +.. note::
> +
> +    In the following code, one line for getting the output port requires some
> +    explanation.

Do we need this note: ?

> +
> +During the initialization process, a static array of destination ports
> +(l2fwd_dst_ports[]) is filled such that for each source port, a destination port
> +is assigned that is either the next or previous enabled port from the portmask.
> +If number of ports are odd in portmask then packet from last port will be
> +forwarded to first port i.e. if portmask=0x07, then forwarding will take place
> +like p0--->p1, p1--->p2, p2--->p0.
> +
> +l2fwd_eventdev_forward() does not stores incoming mbufs. Packet will forwarded
> +be to destination ports via Tx adapter or generic event dev enqueue API
> +depending H/W or S/W scheduler is used.
> +
> +.. code-block:: c
> +
> +        static inline void
> +        l2fwd_eventdev_forward(struct rte_mbuf *m[], uint32_t portid,
> +                               uint16_t nb_rx, uint16_t event_p_id)
> +        {
> +                uint32_t dst_port, i;
> +
> +                dst_port = l2fwd_dst_ports[portid];
> +
> +                for (i = 0; i < nb_rx; i++) {
> +                        if (mac_updating)
> +                                l2fwd_mac_updating(m[i], dst_port);
> +
> +                        m[i]->port = dst_port;
> +                }
> +
> +                if (timer_period > 0) {
> +                        rte_spinlock_lock(&port_stats_lock);
> +                        port_statistics[dst_port].tx += nb_rx;
> +                        rte_spinlock_unlock(&port_stats_lock);
> +                }
> +                /* Registered callback is invoked for Tx */
> +                eventdev_rsrc.send_burst_eventdev(m, nb_rx, event_p_id);
> +        }
> --
> 2.17.1
>