From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
by inbox.dpdk.org (Postfix) with ESMTP id F23D8A0613
for <public@inbox.dpdk.org>; Fri, 27 Sep 2019 17:13:40 +0200 (CEST)
Received: from [92.243.14.124] (localhost [127.0.0.1])
by dpdk.org (Postfix) with ESMTP id 578DA1BEA3;
Fri, 27 Sep 2019 17:13:39 +0200 (CEST)
Received: from mga02.intel.com (mga02.intel.com [134.134.136.20])
by dpdk.org (Postfix) with ESMTP id BFFFF1B9B5
for <dev@dpdk.org>; Fri, 27 Sep 2019 17:13:36 +0200 (CEST)
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from orsmga006.jf.intel.com ([10.7.209.51])
by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;
27 Sep 2019 08:13:35 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.64,555,1559545200"; d="scan'208";a="194513165"
Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203])
by orsmga006.jf.intel.com with ESMTP; 27 Sep 2019 08:13:35 -0700
Received: from HASMSX109.ger.corp.intel.com (10.184.198.21) by
FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS)
id 14.3.439.0; Fri, 27 Sep 2019 08:13:34 -0700
Received: from hasmsx114.ger.corp.intel.com ([169.254.14.116]) by
hasmsx109.ger.corp.intel.com ([169.254.3.38]) with mapi id 14.03.0439.000;
Fri, 27 Sep 2019 18:13:32 +0300
From: "Baran, MarcinX" <marcinx.baran@intel.com>
To: "Richardson, Bruce" <bruce.richardson@intel.com>
CC: "dev@dpdk.org" <dev@dpdk.org>, "Mcnamara, John" <john.mcnamara@intel.com>,
"Kovacevic, Marko" <marko.kovacevic@intel.com>
Thread-Topic: [dpdk-dev] [PATCH v5 6/6] doc/guides/: provide IOAT sample app
guide
Thread-Index: AQHVb4aPH+iU6RkPTUuDqU4O5+UUaKc/XASAgABLLSA=
Date: Fri, 27 Sep 2019 15:13:31 +0000
Message-ID: <06CDC4676D44784DA2DF9423D4B672BE15ECCD7A@HASMSX114.ger.corp.intel.com>
References: <20190919093850.460-1-marcinx.baran@intel.com>
<20190920073714.1314-1-marcinx.baran@intel.com>
<20190920073714.1314-7-marcinx.baran@intel.com>
<20190927132259.GA1859@bricha3-MOBL.ger.corp.intel.com>
In-Reply-To: <20190927132259.GA1859@bricha3-MOBL.ger.corp.intel.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
dlp-product: dlpe-windows
dlp-version: 11.2.0.6
dlp-reaction: no-action
x-originating-ip: [10.184.70.11]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: Re: [dpdk-dev] [PATCH v5 6/6] doc/guides/: provide IOAT sample app
guide
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
-----Original Message-----
From: Bruce Richardson <bruce.richardson@intel.com>=20
Sent: Friday, September 27, 2019 3:23 PM
To: Baran, MarcinX <marcinx.baran@intel.com>
Cc: dev@dpdk.org; Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Mark=
o <marko.kovacevic@intel.com>
Subject: Re: [dpdk-dev] [PATCH v5 6/6] doc/guides/: provide IOAT sample app=
guide
On Fri, Sep 20, 2019 at 09:37:14AM +0200, Marcin Baran wrote:
> Added guide for IOAT sample app usage and code description.
>=20
> Signed-off-by: Marcin Baran <marcinx.baran@intel.com>
> ---
> doc/guides/sample_app_ug/index.rst | 1 +
> doc/guides/sample_app_ug/intro.rst | 4 +
> doc/guides/sample_app_ug/ioat.rst | 764=20
> +++++++++++++++++++++++++++++
> 3 files changed, 769 insertions(+)
> create mode 100644 doc/guides/sample_app_ug/ioat.rst
>=20
<snip>
> +Depending on mode set (whether copy should be done by software or by=20
> +hardware) special structures are assigned to each port. If software=20
> +copy was chosen, application have to assign ring structures for=20
> +packet exchanging between lcores assigned to ports.
> +
> +.. code-block:: c
> +
> + static void
> + assign_rings(void)
> + {
> + uint32_t i;
> +
> + for (i =3D 0; i < cfg.nb_ports; i++) {
> + char ring_name[20];
> +
> + snprintf(ring_name, 20, "rx_to_tx_ring_%u", i);
> + /* Create ring for inter core communication */
> + cfg.ports[i].rx_to_tx_ring =3D rte_ring_create(
> + ring_name, ring_size,
> + rte_socket_id(), RING_F_SP_ENQ);
> +
> + if (cfg.ports[i].rx_to_tx_ring =3D=3D NULL)
> + rte_exit(EXIT_FAILURE, "%s\n",
> + rte_strerror(rte_errno));
> + }
> + }
> +
> +
> +When using hardware copy each Rx queue of the port is assigned an=20
> +IOAT device (``assign_rawdevs()``) using IOAT Rawdev Driver API
> +functions:
> +
> +.. code-block:: c
> +
> + static void
> + assign_rawdevs(void)
> + {
> + uint16_t nb_rawdev =3D 0, rdev_id =3D 0;
> + uint32_t i, j;
> +
> + for (i =3D 0; i < cfg.nb_ports; i++) {
> + for (j =3D 0; j < cfg.ports[i].nb_queues; j++) {
> + struct rte_rawdev_info rdev_info =3D { 0 };
> +
> + do {
> + if (rdev_id =3D=3D rte_rawdev_count())
> + goto end;
> + rte_rawdev_info_get(rdev_id++, &rdev_info);
> + } while (strcmp(rdev_info.driver_name,
> + IOAT_PMD_RAWDEV_NAME_STR) !=3D 0);
> +
> + cfg.ports[i].ioat_ids[j] =3D rdev_id - 1;
> + configure_rawdev_queue(cfg.ports[i].ioat_ids[j]);
> + ++nb_rawdev;
> + }
> + }
> + end:
> + if (nb_rawdev < cfg.nb_ports * cfg.ports[0].nb_queues)
> + rte_exit(EXIT_FAILURE,
> + "Not enough IOAT rawdevs (%u) for all queues (%u).\n",
> + nb_rawdev, cfg.nb_ports * cfg.ports[0].nb_queues);
> + RTE_LOG(INFO, IOAT, "Number of used rawdevs: %u.\n", nb_rawdev);
> + }
> +
> +
> +The initialization of hardware device is done by=20
> +``rte_rawdev_configure()`` function and ``rte_rawdev_info`` struct.
... using ``rte_rawdev_info`` struct
[Marcin] Changed the description accordingly.
> After configuration the device is
> +started using ``rte_rawdev_start()`` function. Each of the above=20
> +operations is done in ``configure_rawdev_queue()``.
In the block below, there is no mention of where dev_config structure comes=
from. Presume it's a global variable, so maybe mention that in the text.
[Marcin] Actually this code snipped was not updated, it now uses local dev_=
config variable of type struct rte_ioat_rawdev_config, so I updated the sni=
ppet.
> +
> +.. code-block:: c
> +
> + static void
> + configure_rawdev_queue(uint32_t dev_id)
> + {
> + struct rte_rawdev_info info =3D { .dev_private =3D &dev_config }=
;
> +
> + /* Configure hardware copy device */
> + dev_config.ring_size =3D ring_size;
> +
> + if (rte_rawdev_configure(dev_id, &info) !=3D 0) {
> + rte_exit(EXIT_FAILURE,
> + "Error with rte_rawdev_configure()\n");
> + }
> + rte_rawdev_info_get(dev_id, &info);
> + if (dev_config.ring_size !=3D ring_size) {
> + rte_exit(EXIT_FAILURE,
> + "Error, ring size is not %d (%d)\n",
> + ring_size, (int)dev_config.ring_size);
> + }
> + if (rte_rawdev_start(dev_id) !=3D 0) {
> + rte_exit(EXIT_FAILURE,
> + "Error with rte_rawdev_start()\n");
> + }
> + }
> +
> +If initialization is successful memory for hardware device statistics=20
> +is allocated.
Missing "," after successful.
Where is this memory allocated? It is done in main or elsewhere?
[Marcin] This comment is invalid now as code has been updated, I removed it=
.
> +
> +Finally ``main()`` functions starts all processing lcores and starts
s/functions/function/
s/processing lcores/packet handling lcores/
[Marcin] Changed the description accordingly
> +printing stats in a loop on master lcore. The application can be
s/master lcore/the master lcore/
[Marcin] Changed the description accordingly
> +interrupted and closed using ``Ctrl-C``. The master lcore waits for=20
> +all slave processes to finish, deallocates resources and exits.
> +
> +The processing lcores launching function are described below.
> +
> +The Lcores Launching Functions
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +As described above ``main()`` function invokes=20
> +``start_forwarding_cores()``
Missing "," after above.
[Marcin] Changed the description accordingly
> +function in order to start processing for each lcore:
> +
> +.. code-block:: c
> +
> + static void start_forwarding_cores(void)
> + {
> + uint32_t lcore_id =3D rte_lcore_id();
> +
> + RTE_LOG(INFO, IOAT, "Entering %s on lcore %u\n",
> + __func__, rte_lcore_id());
> +
> + if (cfg.nb_lcores =3D=3D 1) {
> + lcore_id =3D rte_get_next_lcore(lcore_id, true, true);
> + rte_eal_remote_launch((lcore_function_t *)rxtx_main_loop,
> + NULL, lcore_id);
> + } else if (cfg.nb_lcores > 1) {
> + lcore_id =3D rte_get_next_lcore(lcore_id, true, true);
> + rte_eal_remote_launch((lcore_function_t *)rx_main_loop,
> + NULL, lcore_id);
> +
> + lcore_id =3D rte_get_next_lcore(lcore_id, true, true);
> + rte_eal_remote_launch((lcore_function_t *)tx_main_loop, NULL=
,
> + lcore_id);
> + }
> + }
> +
> +The function launches Rx/Tx processing functions on configured lcores=20
> +for each port using ``rte_eal_remote_launch()``. The configured=20
> +ports,
Remove "for each port"
[Marcin] Removed
> +their number and number of assigned lcores are stored in user-defined=20
> +``rxtx_transmission_config`` struct that is initialized before=20
> +launching
s/is/has been/
Did you describe how that structure was set up previously?
[Marcin] Added description as to how and when it is set up.
> +tasks:
> +
> +.. code-block:: c
> +
> + struct rxtx_transmission_config {
> + struct rxtx_port_config ports[RTE_MAX_ETHPORTS];
> + uint16_t nb_ports;
> + uint16_t nb_lcores;
> + };
> +
> +The Lcores Processing Functions
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +For receiving packets on each port an ``ioat_rx_port()`` function is use=
d.
Missing "," after port.
s/an/the/
[Marcin] Changed the description accordingly
> +The function receives packets on each configured Rx queue. Depending=20
> +on mode
s/mode/the mode/
[Marcin] Changed the description accordingly
> +the user chose, it will enqueue packets to IOAT rawdev channels and=20
> +then invoke copy process (hardware copy), or perform software copy of=20
> +each packet using ``pktmbuf_sw_copy()`` function and enqueue them to 1 r=
te_ring:
s/1 rte_ring/an rte_ring/
[Marcin] Changed the description accordingly
> +
> +.. code-block:: c
> +
> + /* Receive packets on one port and enqueue to IOAT rawdev or rte_rin=
g. */
> + static void
> + ioat_rx_port(struct rxtx_port_config *rx_config)
> + {
> + uint32_t nb_rx, nb_enq, i, j;
> + struct rte_mbuf *pkts_burst[MAX_PKT_BURST];
> + for (i =3D 0; i < rx_config->nb_queues; i++) {
> +
> + nb_rx =3D rte_eth_rx_burst(rx_config->rxtx_port, i,
> + pkts_burst, MAX_PKT_BURST);
> +
> + if (nb_rx =3D=3D 0)
> + continue;
> +
> + port_statistics.rx[rx_config->rxtx_port] +=3D nb_rx;
> +
> + if (copy_mode =3D=3D COPY_MODE_IOAT_NUM) {
> + /* Perform packet hardware copy */
> + nb_enq =3D ioat_enqueue_packets(pkts_burst,
> + nb_rx, rx_config->ioat_ids[i]);
> + if (nb_enq > 0)
> + rte_ioat_do_copies(rx_config->ioat_ids[i]);
> + } else {
> + /* Perform packet software copy, free source packets */
> + int ret;
> + struct rte_mbuf *pkts_burst_copy[MAX_PKT_BURST];
> +
> + ret =3D rte_mempool_get_bulk(ioat_pktmbuf_pool,
> + (void *)pkts_burst_copy, nb_rx);
> +
> + if (unlikely(ret < 0))
> + rte_exit(EXIT_FAILURE,
> + "Unable to allocate memory.\n");
> +
> + for (j =3D 0; j < nb_rx; j++)
> + pktmbuf_sw_copy(pkts_burst[j],
> + pkts_burst_copy[j]);
> +
> + rte_mempool_put_bulk(ioat_pktmbuf_pool,
> + (void *)pkts_burst, nb_rx);
> +
> + nb_enq =3D rte_ring_enqueue_burst(
> + rx_config->rx_to_tx_ring,
> + (void *)pkts_burst_copy, nb_rx, NULL);
> +
> + /* Free any not enqueued packets. */
> + rte_mempool_put_bulk(ioat_pktmbuf_pool,
> + (void *)&pkts_burst_copy[nb_enq],
> + nb_rx - nb_enq);
> + }
> +
> + port_statistics.copy_dropped[rx_config->rxtx_port] +=3D
> + (nb_rx - nb_enq);
> + }
> + }
> +
> +The packets are received in burst mode using ``rte_eth_rx_burst()``=20
> +function. When using hardware copy mode the packets are enqueued in=20
> +copying device's buffer using ``ioat_enqueue_packets()`` which calls=20
> +``rte_ioat_enqueue_copy()``. When all received packets are in the=20
> +buffer the copies are invoked by calling ``rte_ioat_do_copies()``.
s/copies are invoked/copy operations are started/
[Marcin] Changed the description accordingly
> +Function ``rte_ioat_enqueue_copy()`` operates on physical address of=20
> +the packet. Structure ``rte_mbuf`` contains only physical address to=20
> +start of the data buffer (``buf_iova``). Thus the address is shifted
s/shifted/adjusted/
[Marcin] Changed the description accordingly
> +by ``addr_offset`` value in order to get pointer to ``rearm_data``
s/pointer to/the address of/
[Marcin] Changed the description accordingly
> +member of ``rte_mbuf``. That way the packet is copied all at once=20
> +(with data and metadata).
"That way the both the packet data and metadata can be copied in a single o=
peration".
Should also note that this shortcut can be used because the mbufs are "dire=
ct" mbufs allocated by the apps. If another app uses external buffers, or i=
ndirect mbufs, then multiple copy operations must be used.
[Marcin] Changed the description accordingly and added additional informati=
ons.
> +
> +.. code-block:: c
> +
> + static uint32_t
> + ioat_enqueue_packets(struct rte_mbuf **pkts,
> + uint32_t nb_rx, uint16_t dev_id)
> + {
> + int ret;
> + uint32_t i;
> + struct rte_mbuf *pkts_copy[MAX_PKT_BURST];
> +
> + const uint64_t addr_offset =3D RTE_PTR_DIFF(pkts[0]->buf_addr,
> + &pkts[0]->rearm_data);
> +
> + ret =3D rte_mempool_get_bulk(ioat_pktmbuf_pool,
> + (void *)pkts_copy, nb_rx);
> +
> + if (unlikely(ret < 0))
> + rte_exit(EXIT_FAILURE, "Unable to allocate memory.\n");
> +
> + for (i =3D 0; i < nb_rx; i++) {
> + /* Perform data copy */
> + ret =3D rte_ioat_enqueue_copy(dev_id,
> + pkts[i]->buf_iova
> + - addr_offset,
> + pkts_copy[i]->buf_iova
> + - addr_offset,
> + rte_pktmbuf_data_len(pkts[i])
> + + addr_offset,
> + (uintptr_t)pkts[i],
> + (uintptr_t)pkts_copy[i],
> + 0 /* nofence */);
> +
> + if (ret !=3D 1)
> + break;
> + }
> +
> + ret =3D i;
> + /* Free any not enqueued packets. */
> + rte_mempool_put_bulk(ioat_pktmbuf_pool, (void *)&pkts[i], nb_rx =
- i);
> + rte_mempool_put_bulk(ioat_pktmbuf_pool, (void *)&pkts_copy[i],
> + nb_rx - i);
> +
> + return ret;
> + }
> +
> +
> +All done copies are processed by ``ioat_tx_port()`` function. When=20
> +using
s/done/completed/
[Marcin] Changed the description accordingly
> +hardware copy mode the function invokes=20
> +``rte_ioat_completed_copies()`` on each assigned IOAT channel to=20
> +gather copied packets. If software copy mode is used the function=20
> +dequeues copied packets from the rte_ring. Then each packet MAC=20
> +address is changed if it was enabled. After that copies are sent in burs=
t mode using `` rte_eth_tx_burst()``.
> +
> +
> +.. code-block:: c
> +
> + /* Transmit packets from IOAT rawdev/rte_ring for one port. */
> + static void
> + ioat_tx_port(struct rxtx_port_config *tx_config)
> + {
> + uint32_t i, j, nb_dq =3D 0;
> + struct rte_mbuf *mbufs_src[MAX_PKT_BURST];
> + struct rte_mbuf *mbufs_dst[MAX_PKT_BURST];
> +
> + if (copy_mode =3D=3D COPY_MODE_IOAT_NUM) {
> + for (i =3D 0; i < tx_config->nb_queues; i++) {
> + /* Deque the mbufs from IOAT device. */
> + nb_dq =3D rte_ioat_completed_copies(
> + tx_config->ioat_ids[i], MAX_PKT_BURST,
> + (void *)mbufs_src, (void *)mbufs_dst);
> +
> + if (nb_dq =3D=3D 0)
> + break;
> +
> + rte_mempool_put_bulk(ioat_pktmbuf_pool,
> + (void *)mbufs_src, nb_dq);
> +
> + /* Update macs if enabled */
> + if (mac_updating) {
> + for (j =3D 0; j < nb_dq; j++)
> + update_mac_addrs(mbufs_dst[j],
> + tx_config->rxtx_port);
> + }
> +
> + const uint16_t nb_tx =3D rte_eth_tx_burst(
> + tx_config->rxtx_port, 0,
> + (void *)mbufs_dst, nb_dq);
> +
> + port_statistics.tx[tx_config->rxtx_port] +=3D nb_tx;
> +
> + /* Free any unsent packets. */
> + if (unlikely(nb_tx < nb_dq))
> + rte_mempool_put_bulk(ioat_pktmbuf_pool,
> + (void *)&mbufs_dst[nb_tx],
> + nb_dq - nb_tx);
> + }
> + }
> + else {
> + for (i =3D 0; i < tx_config->nb_queues; i++) {
> + /* Deque the mbufs from IOAT device. */
> + nb_dq =3D rte_ring_dequeue_burst(tx_config->rx_to_tx_rin=
g,
> + (void *)mbufs_dst, MAX_PKT_BURST, NULL);
> +
> + if (nb_dq =3D=3D 0)
> + return;
> +
> + /* Update macs if enabled */
> + if (mac_updating) {
> + for (j =3D 0; j < nb_dq; j++)
> + update_mac_addrs(mbufs_dst[j],
> + tx_config->rxtx_port);
> + }
> +
> + const uint16_t nb_tx =3D rte_eth_tx_burst(tx_config->rxt=
x_port,
> + 0, (void *)mbufs_dst, nb_dq);
> +
> + port_statistics.tx[tx_config->rxtx_port] +=3D nb_tx;
> +
> + /* Free any unsent packets. */
> + if (unlikely(nb_tx < nb_dq))
> + rte_mempool_put_bulk(ioat_pktmbuf_pool,
> + (void *)&mbufs_dst[nb_tx],
> + nb_dq - nb_tx);
> + }
> + }
> + }
> +
> +The Packet Copying Functions
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +In order to perform packet copy there is a user-defined function=20
> +``pktmbuf_sw_copy()`` used. It copies a whole packet by copying=20
> +metadata from source packet to new mbuf, and then copying a data=20
> +chunk of source packet. Both memory copies are done using
> +``rte_memcpy()``:
> +
> +.. code-block:: c
> +
> + static inline void
> + pktmbuf_sw_copy(struct rte_mbuf *src, struct rte_mbuf *dst)
> + {
> + /* Copy packet metadata */
> + rte_memcpy(&dst->rearm_data,
> + &src->rearm_data,
> + offsetof(struct rte_mbuf, cacheline1)
> + - offsetof(struct rte_mbuf, rearm_data));
> +
> + /* Copy packet data */
> + rte_memcpy(rte_pktmbuf_mtod(dst, char *),
> + rte_pktmbuf_mtod(src, char *), src->data_len);
> + }
> +
> +The metadata in this example is copied from ``rearm_data`` member of=20
> +``rte_mbuf`` struct up to ``cacheline1``.
> +
> +In order to understand why software packet copying is done as shown=20
> +above please refer to the "Mbuf Library" section of the *DPDK=20
> +Programmer's Guide*.
> \ No newline at end of file
Use a text editor that adds a newline automatically :-)
[Marcin] Added new line at end of file.
/Bruce