From: "Eads, Gage" <gage.eads@intel.com> To: "McDaniel, Timothy" <timothy.mcdaniel@intel.com>, "jerinj@marvell.com" <jerinj@marvell.com> Cc: "mattias.ronnblom@ericsson.com" <mattias.ronnblom@ericsson.com>, "dev@dpdk.org" <dev@dpdk.org>, "Van Haaren, Harry" <harry.van.haaren@intel.com> Subject: Re: [dpdk-dev] [PATCH 05/27] event/dlb: add DLB documentation Date: Thu, 9 Jul 2020 03:29:43 +0000 Message-ID: <DM6PR11MB2572EC3F482E4A06F36420A0F6640@DM6PR11MB2572.namprd11.prod.outlook.com> (raw) In-Reply-To: <1593232671-5690-6-git-send-email-timothy.mcdaniel@intel.com> Hi Tim, > doc/guides/eventdevs/dlb.rst | 497 > ++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 497 insertions(+) > create mode 100644 doc/guides/eventdevs/dlb.rst > > diff --git a/doc/guides/eventdevs/dlb.rst b/doc/guides/eventdevs/dlb.rst new > file mode 100644 index 0000000..21e48fe > --- /dev/null > +++ b/doc/guides/eventdevs/dlb.rst > @@ -0,0 +1,497 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright(c) 2020 Intel Corporation. > + > +Driver for the Intel® Dynamic Load Balancer (DLB) > +================================================== > + > +The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer. > + > +.. note:: > + > + This PMD is disabled by default in the build configuration files, owing to > + an external dependency on the `Netlink Protocol Library Suite > + <http://www.infradead.org/~tgr/libnl/>`_ (libnl-3 and libnl-genl-3) which > + must be installed on the board. Once the Netlink libraries are installed, > + the PMD can be enabled by setting CONFIG_RTE_LIBRTE_PMD_DLB_QM=y > and > + recompiling the DPDK. > + This description appears to be out-of-date. > +Prerequisites > +------------- > + > +- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to > +setup > + the basic DPDK environment. > + > +- Learn about the DLB device and its capabilities at `Intel Support > + <http://www.intel.com/support>`_. FIXME: Add real link when > +documentation > + becomes available. Leftover FIXME > + > +- The DLB kernel module. If it is not included in the machine's OS > + distribution, download it from <FIXME: Add 01.org link when > +available> and > + follow the build instructions. > + Leftover FIXME <snip> > +The hybrid timeout data structures are currently located in > +drivers/event/dlb/dlb_timeout.h: > + > +.. code-block:: c > + > + struct rte_hybrid_timeout_ticks_64 { > + RTE_STD_C11 > + union { > + uint64_t val64; > + struct { > + uint64_t poll_ticks:62; > + uint64_t umonitor_wait:1; > + uint64_t interrupt_wait:1; > + }; > + }; > + }; > + struct rte_hybrid_timeout_ns_32 { > + RTE_STD_C11 > + union { > + uint32_t val32; > + struct { > + uint32_t poll_ns:30; > + uint32_t umonitor_wait:1; > + uint32_t interrupt_wait:1; > + }; > + }; > + }; Is this description correct? dlb_timeout.h isn't introduced in this patchset. > + > +VAS Configuration > +~~~~~~~~~~~~~~~~~ > + > +A VAS is a scheduling domain, of which there are 32 in the DLB. > +(Producer ports in one VAS cannot enqueue events to a different VAS, > +except through the `Data Mover`_.) When a VAS is configured, it I believe this cross-VAS comment is out-of-date. > +allocates load-balanced and directed queues, ports, credits, and other > +hardware resources. Some VAS resource allocations are user-controlled > +-- the number of queues, for example > +-- and others, like credit pools (one directed and one load-balanced > +pool per VAS), are not. > + > +The dlb PMD creates a single VAS per DLB device. Supporting multiple > +VASes per DLB device is a planned feature, where each VAS will be > +represented as a separate event device. Is this comment up-to-date? Patch 16 ("event/dlb: add infos_get and configure") indicates that multiple event devices are supported. <snip> > +Hardware Credits > +~~~~~~~~~~~~~~~~ > + > +DLB uses a hardware credit scheme to prevent software from overflowing > +hardware event storage, with each unit of storage represented by a > +credit. A port spends a credit to enqueue an event, and hardware > +refills the ports with credits as the events are scheduled to ports. > +Refills come from credit pools, and each port is a member of a > +load-balanced credit pool and a directed credit pool. The load-balanced > +credits are used to enqueue to load-balanced queues, and directed credits > are used for directed queues. > + > +An dlb eventdev contains one load-balanced and one directed credit "An dlb" -> "A dlb" > +pool. These pools' sizes are controlled by the nb_events_limit field in > +struct rte_event_dev_config. The load-balanced pool is sized to contain > +nb_events_limit credits, and the directed pool is sized to contain > +nb_events_limit/4 credits. The directed pool size can be overriden with > +the num_dir_credits vdev argument, like so: > + > + .. code-block:: console > + > + --vdev=dlb1_event,num_dir_credits=<value> > + > +This can be used if the default allocation is too low or too high for > +the specific application needs. The PMD also supports a vdev arg that > +limits the max_num_events reported by rte_event_dev_info_get(): > + > + .. code-block:: console > + > + --vdev=dlb1_event,max_num_events=<value> > + > +By default, max_num_events is reported as the total available > +load-balanced credits. If multiple DLB-based applications are being > +used, it may be desirable to control how many load-balanced credits > +each application uses, particularly when application(s) are written to > +configure nb_events_limit equal to the reported max_num_events. > + > +Each port is a member of both credit pools. A port's credit allocation > +is defined by its low watermark, high watermark, and refill quanta. > +These three parameters are calculated by the dlb PMD like so: > + > +- The load-balanced high watermark is set to the port's enqueue_depth. > + The directed high watermark is set to the minimum of the > +enqueue_depth and > + the directed pool size divided by the total number of ports. > +- The refill quanta is set to half the high watermark. > +- The low watermark is set to the minimum of 8 and the refill quanta. From patch 19 ("event/dlb: add port_setup"), this should be 16 instead of 8: + cfg.ldb_credit_quantum = cfg.ldb_credit_high_watermark / 2; + cfg.ldb_credit_low_watermark = RTE_MIN(16, cfg.ldb_credit_quantum); + + cfg.dir_credit_quantum = cfg.dir_credit_high_watermark / 2; + cfg.dir_credit_low_watermark = RTE_MIN(16, cfg.dir_credit_quantum); > + > +When the eventdev is started, each port is pre-allocated a high > +watermark's worth of credits. For example, if an eventdev contains four > +ports with enqueue depths of 32 and a load-balanced credit pool size of > +4096, each port will start with 32 load-balanced credits, and there > +will be 3968 credits available to replenish the ports. Thus, a single > +port is not capable of enqueueing up to the nb_events_limit (without > +any events being dequeued), since the other ports are retaining their > +initial credit allocation; in short, all ports must enqueue in order to reach > the limit. > + > +If a port attempts to enqueue and has no credits available, the enqueue > +operation will fail and the application must retry the enqueue. Credits > +are replenished asynchronously by the DLB hardware. > + <snip> > + > +Ordered Fragments > +~~~~~~~~~~~~~~~~~ > + > +The DLB has a fourth enqueue type: partial enqueue. When a thread is > +processing an ordered event, it can perform up to 16 "partial" > +enqueues, which allows a single received ordered event to result in multiple > reordered events. > + > +For example, consider the case where three events (A, then B, then C) > +are enqueued with ordered scheduling and are received by three different > ports. > +The ports that receive A and C forward events A' and C', while the port > +that receives B generates three partial enqueues -- B1', B2', and B3' > +-- followed by a release operation. The DLB will reorder the events in the > following order: > + > +A', B1', B2', B3', C' > + > +This functionality is not available explicitly through the eventdev > +API, but the dlb PMD provides it through an additional (DLB-specific) > +event operation, RTE_EVENT_DLB_OP_FRAG. I don't believe this OP type appears in this patchset. > + > +Deferred Scheduling > +~~~~~~~~~~~~~~~~~~~ > + > +The DLB PMD's default behavior for managing a CQ is to "pop" the CQ > +once per dequeued event before returning from > +rte_event_dequeue_burst(). This frees the corresponding entries in the > +CQ, which enables the DLB to schedule more events to it. > + > +To support applications seeking finer-grained scheduling control -- for > +example deferring scheduling to get the best possible priority > +scheduling and load-balancing -- the PMD supports a deferred scheduling > +mode. In this mode, the CQ entry is not popped until the *subsequent* > +rte_event_dequeue_burst() call. This mode only applies to load-balanced > +event ports with dequeue depth of 1. > + > +To enable deferred scheduling, use the defer_sched vdev argument like so: > + > + .. code-block:: console > + > + --vdev=dlb1_event,defer_sched=on > + > +Atomic Inflights Allocation > +~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +In the last stage prior to scheduling an atomic event to a CQ, DLB > +holds the inflight event in a temporary buffer that is divided among > +load-balanced queues. If a queue's atomic buffer storage fills up, this > +can result in head-of-line-blocking. For example: > +- An LDB queue allocated N atomic buffer entries > +- All N entries are filled with events from flow X, which is pinned to CQ 0. > + > +Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue > +can be scheduled. The likelihood of this case depends on the eventdev > +configuration, traffic behavior, event processing latency, potential > +for a worker to be interrupted or otherwise delayed, etc. > + > +By default, the PMD allocates 16 buffer entries for each load-balanced > +queue, which provides an even division across all 128 queues but > +potentially wastes buffer space (e.g. if not all queues are used, or > +aren't used for atomic scheduling). > + > +The PMD provides a dev arg to override the default per-queue > +allocation. To increase a vdev's per-queue atomic-inflight allocation to (for > example) 64: > + > + .. code-block:: console > + > + --vdev=dlb1_event,atm_inflights=64 > + This section is duplicated below. > +Atomic Inflights Allocation > +~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +In the last stage prior to scheduling an atomic event to a CQ, DLB > +holds the inflight event in a temporary buffer that is divided among > +load-balanced queues. If a queue's atomic buffer storage fills up, this > +can result in head-of-line-blocking. For example: > +- An LDB queue allocated N atomic buffer entries > +- All N entries are filled with events from flow X, which is pinned to CQ 0. > + > +Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue > +can be scheduled. The likelihood of this case depends on the eventdev > +configuration, traffic behavior, event processing latency, potential > +for a worker to be interrupted or otherwise delayed, etc. > + > +By default, the PMD allocates 16 buffer entries for each load-balanced > +queue, which provides an even division across all 128 queues but > +potentially wastes buffer space (e.g. if not all queues are used, or > +aren't used for atomic scheduling). > + > +The PMD provides a dev arg to override the default per-queue > +allocation. To increase a vdev's per-queue atomic-inflight allocation to (for > example) 64: > + > + .. code-block:: console > + > + --vdev=dlb1_event,atm_inflights=64 > -- > 1.7.10 Thanks, Gage
next prev parent reply other threads:[~2020-07-09 3:29 UTC|newest] Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-06-27 4:37 [dpdk-dev] [PATCH 00/27] event/dlb Intel DLB PMD Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 01/27] eventdev: dlb upstream prerequisites Tim McDaniel 2020-06-27 7:44 ` Jerin Jacob 2020-06-29 19:30 ` McDaniel, Timothy 2020-06-30 4:21 ` Jerin Jacob 2020-06-30 15:37 ` McDaniel, Timothy 2020-06-30 15:57 ` Jerin Jacob 2020-06-30 19:26 ` McDaniel, Timothy 2020-06-30 20:40 ` Pavan Nikhilesh Bhagavatula 2020-06-30 21:07 ` McDaniel, Timothy 2020-07-01 4:50 ` Jerin Jacob 2020-07-01 16:48 ` McDaniel, Timothy 2020-06-30 11:22 ` Kinsella, Ray 2020-06-30 11:30 ` Jerin Jacob 2020-06-30 11:36 ` Kinsella, Ray 2020-06-30 12:14 ` Jerin Jacob 2020-07-02 15:21 ` Kinsella, Ray 2020-07-02 16:35 ` McDaniel, Timothy 2020-06-27 4:37 ` [dpdk-dev] [PATCH 02/27] eventdev: do not pass disable_implicit_release bit to trace macro Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 03/27] event/dlb: add shared code version 10.7.9 Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 04/27] event/dlb: add make and meson build infrastructure Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 05/27] event/dlb: add DLB documentation Tim McDaniel 2020-07-09 3:29 ` Eads, Gage [this message] 2020-06-27 4:37 ` [dpdk-dev] [PATCH 06/27] event/dlb: add dynamic logging Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 07/27] event/dlb: add private data structures and constants Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 08/27] event/dlb: add definitions shared with LKM or shared code Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 09/27] event/dlb: add inline functions used in multiple files Tim McDaniel 2020-07-07 12:02 ` Bruce Richardson 2020-07-07 14:33 ` McDaniel, Timothy 2020-06-27 4:37 ` [dpdk-dev] [PATCH 10/27] event/dlb: add PFPMD-specific interface layer to shared code Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 11/27] event/dlb: add flexible PMD to device interfaces Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 12/27] event/dlb: add the PMD's public interfaces Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 13/27] event/dlb: add xstats support Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 14/27] event/dlb: add PMD self-tests Tim McDaniel 2020-07-10 20:42 ` Eads, Gage 2020-07-29 18:56 ` McDaniel, Timothy 2020-06-27 4:37 ` [dpdk-dev] [PATCH 15/27] event/dlb: add probe Tim McDaniel 2020-07-09 3:45 ` Eads, Gage 2020-06-27 4:37 ` [dpdk-dev] [PATCH 16/27] event/dlb: add infos_get and configure Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 17/27] event/dlb: add queue_def_conf and port_def_conf Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 18/27] event/dlb: add queue setup Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 19/27] event/dlb: add port_setup Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 20/27] event/dlb: add port_link Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 21/27] event/dlb: add queue_release and port_release Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 22/27] event/dlb: add port_unlink and port_unlinks_in_progress Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 23/27] event/dlb: add eventdev_start Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 24/27] event/dlb: add timeout_ticks, dump, xstats, and selftest Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 25/27] event/dlb: add enqueue and its burst variants Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 26/27] event/dlb: add dequeue, dequeue_burst, and variants Tim McDaniel 2020-06-27 4:37 ` [dpdk-dev] [PATCH 27/27] event/dlb: add eventdev_stop and eventdev_close Tim McDaniel [not found] <1593232671-5690-0-git-send-email-timothy.mcdaniel@intel.com> 2020-07-30 19:49 ` [dpdk-dev] [PATCH 00/27] Add Intel DLM PMD to 20.11 McDaniel, Timothy 2020-07-30 19:49 ` [dpdk-dev] [PATCH 05/27] event/dlb: add DLB documentation McDaniel, Timothy 2020-08-11 18:26 ` Jerin Jacob -- strict thread matches above, loose matches on Subject: below -- 2020-06-12 21:24 [dpdk-dev] [PATCH 00/27] V1 event/dlb add Intel DLB PMD McDaniel, Timothy 2020-06-12 21:24 ` [dpdk-dev] [PATCH 05/27] event/dlb: add DLB documentation McDaniel, Timothy
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=DM6PR11MB2572EC3F482E4A06F36420A0F6640@DM6PR11MB2572.namprd11.prod.outlook.com \ --to=gage.eads@intel.com \ --cc=dev@dpdk.org \ --cc=harry.van.haaren@intel.com \ --cc=jerinj@marvell.com \ --cc=mattias.ronnblom@ericsson.com \ --cc=timothy.mcdaniel@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
DPDK patches and discussions This inbox may be cloned and mirrored by anyone: git clone --mirror http://inbox.dpdk.org/dev/0 dev/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 dev dev/ http://inbox.dpdk.org/dev \ dev@dpdk.org public-inbox-index dev Example config snippet for mirrors. Newsgroup available over NNTP: nntp://inbox.dpdk.org/inbox.dpdk.dev AGPL code for this site: git clone https://public-inbox.org/public-inbox.git