From: Spike Du <spiked@nvidia.com>
To: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>,
Matan Azrad <matan@nvidia.com>,
Slava Ovsiienko <viacheslavo@nvidia.com>,
Ori Kam <orika@nvidia.com>,
"NBU-Contact-Thomas Monjalon (EXTERNAL)" <thomas@monjalon.net>,
Wenzhuo Lu <wenzhuo.lu@intel.com>,
Beilei Xing <beilei.xing@intel.com>,
Bernard Iremonger <bernard.iremonger@intel.com>,
Ray Kinsella <mdr@ashroe.eu>, Neil Horman <nhorman@tuxdriver.com>
Cc: "stephen@networkplumber.org" <stephen@networkplumber.org>,
"mb@smartsharesystems.com" <mb@smartsharesystems.com>,
"dev@dpdk.org" <dev@dpdk.org>,
Raslan Darawsheh <rasland@nvidia.com>
Subject: RE: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold
Date: Tue, 7 Jun 2022 06:00:52 +0000 [thread overview]
Message-ID: <MN2PR12MB3647FB20700D0811A0ECCF0FA8A59@MN2PR12MB3647.namprd12.prod.outlook.com> (raw)
In-Reply-To: <de510090-677d-397e-a1df-1c1115108ac6@oktetlabs.ru>
If you think HW knows "emptiness", and we want to stick to DPDK descriptor terms.
Can we use the name "avail_thresh?"
When HW sees available descriptors is under this threshold, an event is triggered.
> -----Original Message-----
> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> Sent: Tuesday, June 7, 2022 1:16 AM
> To: Spike Du <spiked@nvidia.com>; Matan Azrad <matan@nvidia.com>;
> Slava Ovsiienko <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>;
> NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Wenzhuo Lu <wenzhuo.lu@intel.com>; Beilei Xing <beilei.xing@intel.com>;
> Bernard Iremonger <bernard.iremonger@intel.com>; Ray Kinsella
> <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
> Cc: stephen@networkplumber.org; mb@smartsharesystems.com;
> dev@dpdk.org; Raslan Darawsheh <rasland@nvidia.com>
> Subject: Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold
>
> External email: Use caution opening links or attachments
>
>
> On 6/6/22 16:16, Spike Du wrote:
> > Hi Andrew,
> > Please see below for "fill threshold" concept, I'm ok with other
> comments about code.
> >
> > Regards,
> > Spike.
> >
> >
> >> -----Original Message-----
> >> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> >> Sent: Saturday, June 4, 2022 8:46 PM
> >> To: Spike Du <spiked@nvidia.com>; Matan Azrad <matan@nvidia.com>;
> >> Slava Ovsiienko <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>;
> >> NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Wenzhuo
> >> Lu <wenzhuo.lu@intel.com>; Beilei Xing <beilei.xing@intel.com>;
> >> Bernard Iremonger <bernard.iremonger@intel.com>; Ray Kinsella
> >> <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
> >> Cc: stephen@networkplumber.org; mb@smartsharesystems.com;
> >> dev@dpdk.org; Raslan Darawsheh <rasland@nvidia.com>
> >> Subject: Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill
> >> threshold
> >>
> >> External email: Use caution opening links or attachments
> >>
> >>
> >> On 6/3/22 15:48, Spike Du wrote:
> >>> Fill threshold describes the fullness of a Rx queue. If the Rx queue
> >>> fullness is above the threshold, the device will trigger the event
> >>> RTE_ETH_EVENT_RX_FILL_THRESH.
> >>
> >> Sorry, I'm not sure that I understand. As far as I know the process
> >> to add more Rx buffers to Rx queue is called 'refill' in many
> >> drivers. So fill level is a number (or percentage) of free buffers in an Rx
> queue.
> >> If so, fill threashold should be a minimum fill level and below the
> >> level we should generate an event.
> >>
> >> However reading the first paragraph of the descrition it looks like
> >> you mean oposite thing - a number (or percentage) of ready Rx buffers
> >> with received packets.
> >>
> >> I think that the term "fill threshold" is suggested by me, but I did
> >> it with mine understanding of the added feature. Now I'm confused.
> >>
> >> Moreover, I don't understand how "fill threshold" could be in terms
> >> of ready Rx buffers. HW simply don't really know when ready Rx
> >> buffers are processed by SW. So, HW can't say for sure how many ready
> >> Rx buffers are pending. It could be calculated as Rx queue size minus
> >> number of free Rx buffers, but it is imprecise. First of all not all Rx
> descriptors could be used.
> >> Second, HW ring size could differ queue size specified in SW.
> >> Queue size specified in SW could just limit maximum nubmer of free Rx
> >> buffers provided by the driver.
> >>
> >
> > Let me use other terms because "fill"/"refill" is also ambiguous to me.
> > In a RX ring, there are Rx buffers with received packets, you call it
> > "ready Rx buffers", there is a RTE api rte_eth_rx_queue_count() to get the
> number, It's also called "used descriptors" in the code.
> > Also there are Rx buffers provided by SW to allow HW "fill in" received
> packets, we can call it "usable Rx buffers" (here "usable" means usable for
> HW).
>
> May be it is better to stick to Rx descriptor status terminology?
> Available - Rx descriptor available to HW to put received packet to Done - Rx
> descriptor with received packet reported to Sw Unavailable - other (e.g. gap
> which cannot be used or just processed Done, but not refilled (made
> available to HW).
>
> > Let's define Rx queue "fullness":
> > Fullness = ready-Rx-buffers/Rxq-size
>
> i.e. number of DONE descriptors divided by RxQ size
>
> > On the opposite, we have "emptiness"
> > Emptiness = usable-Rx-buffers/Rxq-size
>
> i.e. number of AVAIL descriptors divided by RxQ size Note, that AVAIL !=
> RxQ-size - DONE
>
> HW really knows number of available descriptors by its nature.
> It is a space between latest done and latest received on refill.
>
> HW does not know which descriptors are DONE, since some which are DONE
> before could be already processed by SW, but not yet made available again.
>
>
> > Here "fill threshold" describes "fullness", it's not "refill" described in you
> above words. Because in your words, "refill" is the opposite, it's filling
> "usable/free Rx buffers", or "emptiness".
> >
> > I can only briefly explain how mlx5 works to get LWM, because I'm not a
> Firmware guy.
> > Mlx5 Rx queue is basically RDMA queue. It has two indexes: producer index
> which increases when HW fills in packet, consumer index which increases
> when SW consumes the packet.
> > The queue size is known when it's created. The fullness is something like
> (producer_index - consumer_index) (I don't consider in wrap-around here).
> > So mlx5 has the way to get the fullness or emptiness in HW or FW.
> > Another detail is mlx5 uses the term "LWM"(limit watermark), which
> describes "emptiness". When usable-Rx-buffers is below LWM, we trigger an
> event.
> > But Thomas think "fullness" is easier to understand, so we use "fullness" in
> rte APIs and we'll translate it to LWM in mlx5 PMD.
>
> HW simply does now know fullness and there can't generate any events
> based on it. It is a problem on Rx when there is now available descriptors. I.e.
> emptiness.
>
> >>> Fill threshold is defined as a percentage of Rx queue size with
> >>> valid value of [0,99].
> >>> Setting fill threshold to 0 means disable it, which is the default.
> >>> Add fill threshold configuration and query driver callbacks in
> eth_dev_ops.
> >>> Add command line options to support fill_thresh per-rxq configure.
> >>> - Command syntax:
> >>> set port <port_id> rxq <rxq_id> fill_thresh <fill_thresh_num>
> >>>
> >>> - Example commands:
> >>> To configure fill_thresh as 30% of rxq size on port 1 rxq 0:
> >>> testpmd> set port 1 rxq 0 fill_thresh 30
> >>>
> >>> To disable fill_thresh on port 1 rxq 0:
> >>> testpmd> set port 1 rxq 0 fill_thresh 0
> >>>
> >>> Signed-off-by: Spike Du <spiked@nvidia.com>
>
> [snip]
next prev parent reply other threads:[~2022-06-07 6:00 UTC|newest]
Thread overview: 131+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-01 3:22 [RFC 0/6] net/mlx5: introduce limit watermark and host shaper Spike Du
2022-04-01 3:22 ` [RFC 1/6] net/mlx5: add LWM support for Rxq Spike Du
2022-05-06 3:56 ` [RFC v1 0/7] net/mlx5: introduce limit watermark and host shaper Spike Du
2022-05-06 3:56 ` [RFC v1 1/7] net/mlx5: add LWM support for Rxq Spike Du
2022-05-06 3:56 ` [RFC v1 2/7] common/mlx5: share interrupt management Spike Du
2022-05-06 3:56 ` [RFC v1 3/7] ethdev: introduce Rx queue based limit watermark Spike Du
2022-05-19 9:37 ` Andrew Rybchenko
2022-05-06 3:56 ` [RFC v1 4/7] net/mlx5: add LWM event handling support Spike Du
2022-05-06 3:56 ` [RFC v1 5/7] net/mlx5: support Rx queue based limit watermark Spike Du
2022-05-06 3:56 ` [RFC v1 6/7] net/mlx5: add private API to config host port shaper Spike Du
2022-05-06 3:56 ` [RFC v1 7/7] app/testpmd: add LWM and Host Shaper command Spike Du
2022-05-22 5:58 ` [RFC v2 0/7] introduce per-queue limit watermark and host shaper Spike Du
2022-05-22 5:58 ` [RFC v2 1/7] net/mlx5: add LWM support for Rxq Spike Du
2022-05-22 5:58 ` [RFC v2 2/7] common/mlx5: share interrupt management Spike Du
2022-05-22 5:58 ` [RFC v2 3/7] ethdev: introduce Rx queue based limit watermark Spike Du
2022-05-22 15:23 ` Stephen Hemminger
2022-05-23 3:01 ` Spike Du
2022-05-23 21:45 ` Thomas Monjalon
2022-05-24 2:50 ` Spike Du
2022-05-24 8:18 ` Thomas Monjalon
2022-05-25 12:59 ` Andrew Rybchenko
2022-05-25 13:58 ` Thomas Monjalon
2022-05-25 14:23 ` Andrew Rybchenko
2022-05-23 22:54 ` Stephen Hemminger
2022-05-24 3:46 ` Spike Du
2022-05-22 15:24 ` Stephen Hemminger
2022-05-23 2:18 ` Spike Du
2022-05-23 6:07 ` Morten Brørup
2022-05-23 10:58 ` Thomas Monjalon
2022-05-23 14:10 ` Spike Du
2022-05-23 14:39 ` Thomas Monjalon
2022-05-24 6:35 ` Andrew Rybchenko
2022-05-24 9:40 ` Morten Brørup
2022-05-22 5:58 ` [RFC v2 4/7] net/mlx5: add LWM event handling support Spike Du
2022-05-22 5:58 ` [RFC v2 5/7] net/mlx5: support Rx queue based limit watermark Spike Du
2022-05-22 5:58 ` [RFC v2 6/7] net/mlx5: add private API to config host port shaper Spike Du
2022-05-22 5:59 ` [RFC v2 7/7] app/testpmd: add LWM and Host Shaper command Spike Du
2022-05-24 15:20 ` [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Spike Du
2022-05-24 15:20 ` [PATCH v3 1/7] net/mlx5: add LWM support for Rxq Spike Du
2022-05-24 15:20 ` [PATCH v3 2/7] common/mlx5: share interrupt management Spike Du
2022-05-24 15:20 ` [PATCH v3 3/7] ethdev: introduce Rx queue based limit watermark Spike Du
2022-05-24 15:20 ` [PATCH v3 4/7] net/mlx5: add LWM event handling support Spike Du
2022-05-24 15:20 ` [PATCH v3 5/7] net/mlx5: support Rx queue based limit watermark Spike Du
2022-05-24 15:20 ` [PATCH v3 6/7] net/mlx5: add private API to config host port shaper Spike Du
2022-05-24 15:20 ` [PATCH v3 7/7] app/testpmd: add LWM and Host Shaper command Spike Du
2022-05-24 15:59 ` [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Thomas Monjalon
2022-05-24 19:00 ` Morten Brørup
2022-05-24 19:22 ` Thomas Monjalon
2022-05-25 14:11 ` Andrew Rybchenko
2022-05-25 13:14 ` Spike Du
2022-05-25 13:40 ` Morten Brørup
2022-05-25 13:59 ` Spike Du
2022-05-25 14:16 ` Morten Brørup
2022-05-25 14:30 ` Andrew Rybchenko
2022-06-03 12:48 ` [PATCH v4 0/7] introduce per-queue fill threshold " Spike Du
2022-06-03 12:48 ` [PATCH v4 1/7] net/mlx5: add LWM support for Rxq Spike Du
2022-06-03 12:48 ` [PATCH v4 2/7] common/mlx5: share interrupt management Spike Du
2022-06-03 14:30 ` Ray Kinsella
2022-06-03 12:48 ` [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold Spike Du
2022-06-03 14:30 ` Ray Kinsella
2022-06-04 12:46 ` Andrew Rybchenko
2022-06-06 13:16 ` Spike Du
2022-06-06 17:15 ` Andrew Rybchenko
2022-06-06 21:30 ` Thomas Monjalon
2022-06-07 8:02 ` Andrew Rybchenko
2022-06-07 6:00 ` Spike Du [this message]
2022-06-06 15:49 ` Stephen Hemminger
2022-06-03 12:48 ` [PATCH v4 4/7] net/mlx5: add LWM event handling support Spike Du
2022-06-03 12:48 ` [PATCH v4 5/7] net/mlx5: support Rx queue based fill threshold Spike Du
2022-06-03 12:48 ` [PATCH v4 6/7] net/mlx5: add private API to config host port shaper Spike Du
2022-06-03 14:55 ` Ray Kinsella
2022-06-03 12:48 ` [PATCH v4 7/7] app/testpmd: add Host Shaper command Spike Du
2022-06-07 12:59 ` [PATCH v5 0/7] introduce per-queue available descriptor threshold and host shaper Spike Du
2022-06-07 12:59 ` [PATCH v5 1/7] net/mlx5: add LWM support for Rxq Spike Du
2022-06-08 20:10 ` Matan Azrad
2022-06-07 12:59 ` [PATCH v5 2/7] common/mlx5: share interrupt management Spike Du
2022-06-07 12:59 ` [PATCH v5 3/7] ethdev: introduce Rx queue based available descriptor threshold Spike Du
2022-06-07 12:59 ` [PATCH v5 4/7] net/mlx5: add LWM event handling support Spike Du
2022-06-07 12:59 ` [PATCH v5 5/7] net/mlx5: support Rx queue based available descriptor threshold Spike Du
2022-06-07 12:59 ` [PATCH v5 6/7] net/mlx5: add private API to config host port shaper Spike Du
2022-06-07 12:59 ` [PATCH v5 7/7] app/testpmd: add Host Shaper command Spike Du
2022-06-09 7:55 ` Andrew Rybchenko
2022-06-10 2:22 ` Spike Du
2022-06-13 2:50 ` [PATCH v6] " Spike Du
2022-06-13 2:50 ` Spike Du
2022-06-14 9:43 ` Singh, Aman Deep
2022-06-14 9:54 ` Spike Du
2022-06-14 12:01 ` [PATCH v7] " Spike Du
2022-06-14 12:01 ` Spike Du
2022-06-15 7:51 ` Matan Azrad
2022-06-15 11:08 ` Thomas Monjalon
2022-06-15 12:58 ` [PATCH v8 0/6] introduce per-queue available descriptor threshold and host shaper Spike Du
2022-06-15 12:58 ` [PATCH v8 1/6] net/mlx5: add LWM support for Rxq Spike Du
2022-06-15 14:43 ` [PATCH v9 0/6] introduce per-queue available descriptor threshold and host shaper Spike Du
2022-06-15 14:43 ` [PATCH v9 1/6] net/mlx5: add LWM support for Rxq Spike Du
2022-06-16 8:41 ` [PATCH v10 0/6] introduce per-queue available descriptor threshold and host shaper Spike Du
2022-06-16 8:41 ` [PATCH v10 1/6] net/mlx5: add LWM support for Rxq Spike Du
2022-06-16 8:41 ` [PATCH v10 2/6] common/mlx5: share interrupt management Spike Du
2022-06-23 16:05 ` Ray Kinsella
2022-06-16 8:41 ` [PATCH v10 3/6] net/mlx5: add LWM event handling support Spike Du
2022-06-16 8:41 ` [PATCH v10 4/6] net/mlx5: support Rx queue based available descriptor threshold Spike Du
2022-06-16 8:41 ` [PATCH v10 5/6] net/mlx5: add private API to config host port shaper Spike Du
2022-06-16 8:41 ` [PATCH v10 6/6] app/testpmd: add Host Shaper command Spike Du
2022-06-19 8:14 ` [PATCH v10 0/6] introduce per-queue available descriptor threshold and host shaper Raslan Darawsheh
2022-06-15 14:43 ` [PATCH v9 2/6] common/mlx5: share interrupt management Spike Du
2022-06-15 14:43 ` [PATCH v9 3/6] net/mlx5: add LWM event handling support Spike Du
2022-06-15 14:43 ` [PATCH v9 4/6] net/mlx5: support Rx queue based available descriptor threshold Spike Du
2022-06-15 14:43 ` [PATCH v9 5/6] net/mlx5: add private API to config host port shaper Spike Du
2022-06-15 14:43 ` [PATCH v9 6/6] app/testpmd: add Host Shaper command Spike Du
2022-06-15 12:58 ` [PATCH v8 2/6] common/mlx5: share interrupt management Spike Du
2022-06-15 12:58 ` [PATCH v8 3/6] net/mlx5: add LWM event handling support Spike Du
2022-06-15 12:58 ` [PATCH v8 4/6] net/mlx5: support Rx queue based available descriptor threshold Spike Du
2022-06-15 12:58 ` [PATCH v8 5/6] net/mlx5: add private API to config host port shaper Spike Du
2022-06-15 12:58 ` [PATCH v8 6/6] app/testpmd: add Host Shaper command Spike Du
2022-06-08 9:43 ` [PATCH v5 0/7] introduce per-queue available descriptor threshold and host shaper Andrew Rybchenko
2022-06-08 16:35 ` [PATCH v6] ethdev: introduce available Rx descriptors threshold Andrew Rybchenko
2022-06-08 17:22 ` Thomas Monjalon
2022-06-08 17:46 ` Thomas Monjalon
2022-06-09 0:17 ` fengchengwen
2022-06-09 7:05 ` Thomas Monjalon
2022-06-10 0:01 ` fengchengwen
2022-04-01 3:22 ` [RFC 2/6] common/mlx5: share interrupt management Spike Du
2022-04-01 3:22 ` [RFC 3/6] net/mlx5: add LWM event handling support Spike Du
2022-04-01 3:22 ` [RFC 4/6] net/mlx5: add private API to configure Rxq LWM Spike Du
2022-04-01 3:22 ` [RFC 5/6] net/mlx5: add private API to config host port shaper Spike Du
2022-04-01 3:22 ` [RFC 6/6] app/testpmd: add LWM and Host Shaper command Spike Du
2022-04-05 8:58 ` [RFC 0/6] net/mlx5: introduce limit watermark and host shaper Jerin Jacob
2022-04-26 2:42 ` Spike Du
2022-05-01 12:50 ` Jerin Jacob
2022-05-02 3:58 ` Spike Du
2022-04-29 5:48 ` Spike Du
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=MN2PR12MB3647FB20700D0811A0ECCF0FA8A59@MN2PR12MB3647.namprd12.prod.outlook.com \
--to=spiked@nvidia.com \
--cc=andrew.rybchenko@oktetlabs.ru \
--cc=beilei.xing@intel.com \
--cc=bernard.iremonger@intel.com \
--cc=dev@dpdk.org \
--cc=matan@nvidia.com \
--cc=mb@smartsharesystems.com \
--cc=mdr@ashroe.eu \
--cc=nhorman@tuxdriver.com \
--cc=orika@nvidia.com \
--cc=rasland@nvidia.com \
--cc=stephen@networkplumber.org \
--cc=thomas@monjalon.net \
--cc=viacheslavo@nvidia.com \
--cc=wenzhuo.lu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).