From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 94170A2EEB for ; Fri, 13 Sep 2019 11:25:03 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5408F1F095; Fri, 13 Sep 2019 11:25:02 +0200 (CEST) Received: from mail-io1-f66.google.com (mail-io1-f66.google.com [209.85.166.66]) by dpdk.org (Postfix) with ESMTP id 2DD0B1F077 for ; Fri, 13 Sep 2019 11:25:00 +0200 (CEST) Received: by mail-io1-f66.google.com with SMTP id b136so61638606iof.3 for ; Fri, 13 Sep 2019 02:25:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=h/bzVmTrYEEqh0ySaf6kfoK2Wn22n07iXt8FM2tOE4c=; b=av5aZGPoQk2CktbqctL7e+IBBBMBFaJI5s1bqnmcm1FLWDUtk4aBOBj6ePctaRQioI PO16SZ9MBivaXfNzN0fJ3ruOw3mQwwfNg6kInzmspffrtMyeAklU11uzE0B/Jw4dOlig KWSTlgms6/jt7Y1yxYqOHoIB4uwNeFIDvbLV1w1xriGZdZeT2ZvjuKFOxUlz85+Gww83 XvDTB69IAZjFzspnGGxiiGaZChRilraZWtrEHyk0S+UvgX9/m1fPX6dg58I1YRw8AmJ4 ft6Z9okkIihEoW8F7sRqTjEJW/rHSEB9bzFHXwiId1pXUUmN8T2hLFKAXJmkWaxoHSmm gZ5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=h/bzVmTrYEEqh0ySaf6kfoK2Wn22n07iXt8FM2tOE4c=; b=hvOXn2pTSrHIOMkmB/dBlfIhCEciB9KLdIGn0ymMEE6F+gGjsRJeNs3MMFKcadOvLF abHMQINnXc9ByTj1iLN0GtTLqvX6Oe93QrUocLDYZRm+PqiYqAsrzzLh/0zdoVW1hNVE 7NokskpxAO66dyZOz4Hw70yKDTUnQv1CSelveIPQ4u2biSVTyydzocyCh9Z1fcXOYV9C Tiks5UJQd/m6i0LOvG3+fXGytfJCr+VA7PQ/55XADdTrywcGxiT/7M06c2mJmOgq9QoY NHAYlLRl9hjfeWqktUWlIkc8WlIIR8uY30pc8HNpJm+b2FA7rfMRNwccWCqWrQJnr7u8 d+dQ== X-Gm-Message-State: APjAAAVEInQ8GbLTHFtDwX+QNxYtsqOMsGoNk5sOulzlhp2fHM0VRMBd 0y7fnRV4uQLJEfJHhq8mfiCo41olpjjYW7aB5rlCBmiYM0TKgw== X-Google-Smtp-Source: APXvYqxEJB89JkQuZCkZNIaA7bG68AJAOuBvsyuHlv/jp9RQJsaaUEuFo2QIUutWHNjuilZc/ViUX6KlgQU9OTFF51M= X-Received: by 2002:a6b:c895:: with SMTP id y143mr7509106iof.271.1568366699142; Fri, 13 Sep 2019 02:24:59 -0700 (PDT) MIME-Version: 1.0 References: <1567758021-8631-1-git-send-email-skori@marvell.com> <1567758021-8631-2-git-send-email-skori@marvell.com> In-Reply-To: <1567758021-8631-2-git-send-email-skori@marvell.com> From: Jerin Jacob Date: Fri, 13 Sep 2019 14:54:47 +0530 Message-ID: To: Sunil Kumar Kori Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH] examples/l2fwd-event: add l2fwd with eventdev mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Fri, Sep 6, 2019 at 1:50 PM Sunil Kumar Kori wrote: > > Patchset adds a new application to demonstrate usage of poll > and eventdev mode. > > Following is the summary of newly added features: > 1. Exposing following new command line parameters. > - mode: It dictates the mode of operation i.e. poll or eventdev. > - eventq_sync: It dictates eventq synchronization method i.e. > atomic or ordered. Currently only atomic and > ordered methods are implemented. > > 2. By default, application will be working into eventdev mode. > 3. All the eventdev resources are initialized with default values. > no configuration are exposed to the user. > > Following is the summary of default configuration: > - Single instance of eventdev supported. > - Number of event ports are equal to number of worker thread. > - Number of event queue are equal number of ethernet ports. > - Each event port is linked to all existing event queues. > - Dedicated Rx adapter for each Ethernet port and all Ethernet > port Rx queues are added to respective Rx adapter. > - Dedicated Tx adapter for each Ethernet port and all Ethernet > port Rx queues are added to respective Rx adapter. > > Signed-off-by: Sunil Kumar Kori > --- > examples/Makefile | 1 + > examples/l2fwd-event/Makefile | 57 +++ > examples/l2fwd-event/l2fwd_common.h | 46 ++ > examples/l2fwd-event/l2fwd_eventdev.c | 686 ++++++++++++++++++++++++++++++ > examples/l2fwd-event/l2fwd_eventdev.h | 82 ++++ > examples/l2fwd-event/main.c | 771 ++++++++++++++++++++++++++++++++++ > examples/l2fwd-event/meson.build | 12 + # Please add the documentation # Split this patch to more logical patches # Update the maintainers file # Make sure it works with both eventdev HW and SW PMD. > 7 files changed, 1655 insertions(+) > create mode 100644 examples/l2fwd-event/Makefile > create mode 100644 examples/l2fwd-event/l2fwd_common.h > create mode 100644 examples/l2fwd-event/l2fwd_eventdev.c > create mode 100644 examples/l2fwd-event/l2fwd_eventdev.h > create mode 100644 examples/l2fwd-event/main.c > create mode 100644 examples/l2fwd-event/meson.build > > diff --git a/examples/l2fwd-event/l2fwd_common.h b/examples/l2fwd-event/l2fwd_common.h > new file mode 100644 > index 0000000..28919e4 > --- /dev/null > +++ b/examples/l2fwd-event/l2fwd_common.h > @@ -0,0 +1,46 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(C) 2019 Marvell International Ltd. > + */ > + > +#ifndef __L2FWD_COMMON_H__ > +#define __L2FWD_COMMON_H__ > + > +#define MAX_PKT_BURST 32 > +#define MAX_RX_QUEUE_PER_LCORE 16 > +#define MAX_TX_QUEUE_PER_PORT 16 > + > +#define RTE_LOGTYPE_L2FWD RTE_LOGTYPE_USER1 > + > +struct lcore_queue_conf { > + uint32_t rx_port_list[MAX_RX_QUEUE_PER_LCORE]; > + uint32_t n_rx_port; > +} __rte_cache_aligned; > + > +/* Per-port statistics struct */ > +struct l2fwd_port_statistics { > + uint64_t dropped; > + uint64_t tx; > + uint64_t rx; > +} __rte_cache_aligned; > + > +extern struct rte_mempool *l2fwd_pktmbuf_pool; > + > +extern struct rte_ether_addr l2fwd_ports_eth_addr[RTE_MAX_ETHPORTS]; > + > +extern uint32_t l2fwd_enabled_port_mask; > + > +extern int mac_updating; > + > +extern uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS]; > + > +extern struct l2fwd_port_statistics port_statistics[RTE_MAX_ETHPORTS]; > + > +extern volatile bool force_quit; > + > +extern uint64_t timer_period; Remove all the global variable and move to common structure. > + > +void l2fwd_mac_updating(struct rte_mbuf *m, uint32_t dest_portid); > + > +void print_stats(void); > + > +#endif /* __L2FWD_EVENTDEV_H__ */ > diff --git a/examples/l2fwd-event/l2fwd_eventdev.c b/examples/l2fwd-event/l2fwd_eventdev.c > new file mode 100644 > index 0000000..744040e > --- /dev/null > +++ b/examples/l2fwd-event/l2fwd_eventdev.c > @@ -0,0 +1,686 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(C) 2019 Marvell International Ltd. > + */ > + > +#include > +#include > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "l2fwd_common.h" > +#include "l2fwd_eventdev.h" > + > +enum { > + CMD_LINE_OPT_MODE_NUM = 265, > + CMD_LINE_OPT_EVENTQ_SYNC_NUM, > +}; > + > +static const struct option eventdev_lgopts[] = { > + {CMD_LINE_OPT_MODE, 1, 0, CMD_LINE_OPT_MODE_NUM}, > + {CMD_LINE_OPT_EVENTQ_SYNC, 1, 0, CMD_LINE_OPT_EVENTQ_SYNC_NUM}, > + {NULL, 0, 0, 0} > +}; > + > +/* Eventdev command line options */ > +int evd_argc; > +char *evd_argv[3]; > + > +/* Default configurations */ > +int pkt_transfer_mode = PACKET_TRANSFER_MODE_EVENTDEV; > +int eventq_sync_mode = RTE_SCHED_TYPE_ATOMIC; > +uint32_t num_workers = RTE_MAX_LCORE; > +struct eventdev_resources eventdev_rsrc; Remove global variables. > +static struct rte_eth_conf port_config = { > + .rxmode = { > + .mq_mode = ETH_MQ_RX_RSS, > + .max_rx_pkt_len = RTE_ETHER_MAX_LEN, > + .split_hdr_size = 0, > + .offloads = DEV_RX_OFFLOAD_CHECKSUM > + }, > + .rx_adv_conf = { > + .rss_conf = { > + .rss_key = NULL, > + .rss_hf = ETH_RSS_IP, RSS not required here. I understand, SW Rx adapter need mbuf.hash so select the same based on internal port avilable or not? > + } > + }, > + .txmode = { > + .mq_mode = ETH_MQ_TX_NONE, > + } > +}; > + > +static struct rte_event_dev_config event_d_conf = { > + .nb_event_queues = 1, > + .nb_event_ports = RTE_MAX_LCORE, > + .nb_events_limit = 4096, > + .nb_event_queue_flows = 1024, > + .nb_event_port_dequeue_depth = 128, > + .nb_event_port_enqueue_depth = 128 > +}; > + > +static struct rte_event_port_conf event_p_conf = { > + .dequeue_depth = 32, > + .enqueue_depth = 32, > + .new_event_threshold = 4096 > +}; > + > +static struct rte_event_queue_conf event_q_conf = { > + .nb_atomic_flows = 1024, > + .nb_atomic_order_sequences = 1024, > + .event_queue_cfg = 0, > + .schedule_type = RTE_SCHED_TYPE_ATOMIC, > + .priority = RTE_EVENT_DEV_PRIORITY_HIGHEST > +}; > + > +static struct rte_event_eth_rx_adapter_queue_conf eth_q_conf = { > + .rx_queue_flags = 0, > + .servicing_weight = 1, > + .ev = { > + .queue_id = 0, > + .priority = RTE_EVENT_DEV_PRIORITY_HIGHEST, > + .sched_type = RTE_SCHED_TYPE_ATOMIC > + } > +}; This all thing can be local to that function in the stack. > + > +/* Send burst of packets on an output interface */ > +static inline int send_burst_eventdev_generic(struct rte_mbuf *m[], uint16_t n, > + uint16_t event_p_id) > +{ > + struct rte_event events[MAX_PKT_BURST]; > + uint8_t event_d_id; > + int ret, i; > + > + event_d_id = eventdev_rsrc.event_d_id; Remove the access to global variable. > + > + for (i = 0; i < n; i++) { > + events[i].queue_id = 0; It should be Tx queue. > + events[i].op = RTE_EVENT_OP_FORWARD; > + events[i].mbuf = m[i]; > + } > + > + ret = rte_event_enqueue_burst(event_d_id, event_p_id, events, n); > + if (unlikely(ret < n)) { > + do { > + rte_pktmbuf_free(m[ret]); > + } while (++ret < n); > + } > + > + return 0; > +} > + > +/* Send burst of packets on an output interface */ > +static inline int send_burst_eventdev_adapter(struct rte_mbuf *m[], uint16_t n, > + uint16_t event_p_id) > +{ > + struct rte_event events[MAX_PKT_BURST]; > + uint8_t event_d_id; > + int32_t ret, i; > + > + event_d_id = eventdev_rsrc.event_d_id; > + > + for (i = 0; i < n; i++) { > + events[i].queue_id = 0; > + events[i].op = RTE_EVENT_OP_FORWARD; > + events[i].mbuf = m[i]; > + rte_event_eth_tx_adapter_txq_set(events[i].mbuf, 0); > + } A lot of common code between send_burst_eventdev_generic() and this function. Please use compile-time builtin constant scheme create runtime workers. > + > + ret = rte_event_eth_tx_adapter_enqueue(event_d_id, event_p_id, > + events, n); > + if (unlikely(ret < n)) { > + do { > + rte_pktmbuf_free(m[ret]); > + } while (++ret < n); > + } > + > + return 0; > +} > + > +static void > +l2fwd_eventdev_forward(struct rte_mbuf *m[], uint32_t portid, > + uint16_t nb_rx, uint16_t event_p_id) > +{ > + uint32_t dst_port, i; > + > + dst_port = l2fwd_dst_ports[portid]; > + > + for (i = 0; i < nb_rx; i++) { > + if (mac_updating) > + l2fwd_mac_updating(m[i], dst_port); > + > + m[i]->port = dst_port; > + } > + > + port_statistics[dst_port].tx += nb_rx; > + eventdev_rsrc.send_burst_eventdev(m, nb_rx, event_p_id); Remove this function pointer scheme. > +} > + > +/* main eventdev processing loop */ > +void l2fwd_main_loop_eventdev(void) Have seperate worker for burst and non burst. > +{ > + uint64_t prev_tsc, diff_tsc, cur_tsc, timer_tsc; > + struct rte_event events[MAX_PKT_BURST]; > + struct rte_mbuf *mbuf[MAX_PKT_BURST]; > + uint32_t i, j = 0, nb_rx; > + uint16_t event_d_id; > + uint16_t event_p_id; > + uint32_t lcore_id; > + uint16_t deq_len; > + uint32_t portid; > + > + prev_tsc = 0; > + timer_tsc = 0; > + > + /* Assign dedicated event port for enqueue/dequeue operation */ > + deq_len = event_d_conf.nb_event_port_dequeue_depth; > + event_d_id = eventdev_rsrc.event_d_id; > + event_p_id = get_free_event_port(); > + lcore_id = rte_lcore_id(); > + > + RTE_LOG(INFO, L2FWD, "entering eventdev main loop on lcore %u\n", > + lcore_id); > + > + while (!force_quit) { > + > + /* if timer is enabled */ > + if (timer_period > 0) { > + cur_tsc = rte_rdtsc(); > + diff_tsc = cur_tsc - prev_tsc; > + > + /* advance the timer */ > + timer_tsc += diff_tsc; > + > + /* if timer has reached its timeout */ > + if (unlikely(timer_tsc >= timer_period)) { > + > + /* do this only on master core */ > + if (lcore_id == rte_get_master_lcore()) { > + print_stats(); > + > + /* reset the timer */ > + timer_tsc = 0; > + } > + } > + prev_tsc = cur_tsc; > + } > + > + /* Read packet from eventdev */ > + nb_rx = rte_event_dequeue_burst(event_d_id, event_p_id, > + events, deq_len, 0); > + if (nb_rx == 0) { > + rte_pause(); > + continue; > + } > + > + for (i = 0; i < nb_rx; i++) { > + mbuf[i] = events[i].mbuf; > + rte_prefetch0(rte_pktmbuf_mtod(mbuf[i], void *)); Please prefetch ahead. > + } > + > + portid = mbuf[0]->port; > + port_statistics[portid].rx++; > + for (i = 1; i < nb_rx; i++) { > + if (portid != mbuf[i]->port) { > + l2fwd_eventdev_forward(&mbuf[j], portid, i - j, > + event_p_id); > + j = i; > + portid = mbuf[i]->port; > + } > + port_statistics[portid].rx++; > + } > + > + /* Send remaining packets */ > + l2fwd_eventdev_forward(&mbuf[j], portid, i - j, event_p_id); > + } > +} > + > +#define EVENT_DEV_PARAM_PRESENT 0x8000 /* Random value*/ I think, we can remove this. > + > +/* Packet transfer mode of the application */ > +#define PACKET_TRANSFER_MODE_POLL 1 > +#define PACKET_TRANSFER_MODE_EVENTDEV 2 > + > +#define CMD_LINE_OPT_MODE "mode" > +#define CMD_LINE_OPT_EVENTQ_SYNC "eventq-sync" > + > +typedef int (*tx_eventdev_t)(struct rte_mbuf *m[], uint16_t n, uint16_t port); > + > +struct eventdev_queues { > + uint8_t *event_q_id; > + uint8_t nb_queues; > +}; > + > +struct eventdev_ports { > + uint8_t *event_p_id; > + uint8_t nb_ports; > + rte_spinlock_t lock; > +}; > + > +struct eventdev_rx_adptr { > + uint8_t nb_rx_adptr; > + uint8_t *rx_adptr; > +}; > + > +struct eventdev_tx_adptr { > + uint8_t nb_tx_adptr; > + uint8_t *tx_adptr; > +}; > + > +struct eventdev_resources { > + tx_eventdev_t send_burst_eventdev; > + struct eventdev_rx_adptr rx_adptr; > + struct eventdev_tx_adptr tx_adptr; > + struct eventdev_queues evq; > + struct eventdev_ports evp; > + uint8_t event_d_id; > +}; > + > +extern int pkt_transfer_mode; > +extern int eventq_sync_mode; > +extern struct eventdev_resources eventdev_rsrc; > +extern int evd_argc; > +extern char *evd_argv[3]; Remove global variables. > + > +/* Event device and required resource setup function */ > +int eventdev_resource_setup(int argc, char **argv); > + > +/* Returns next available event port */ > +int get_free_event_port(void); > + > +/* Event processing function */ > +void l2fwd_main_loop_eventdev(void); > + > +#endif /* __L2FWD_EVENTDEV_H__ */ > +