From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
To: "Burakov, Anatoly" <anatoly.burakov@intel.com>,
"dev@dpdk.org" <dev@dpdk.org>,
"Hunt, David" <david.hunt@intel.com>
Cc: "Loftus, Ciara" <ciara.loftus@intel.com>
Subject: Re: [dpdk-dev] [PATCH v6 5/7] power: support callbacks for multiple Rx queues
Date: Wed, 7 Jul 2021 17:09:39 +0000 [thread overview]
Message-ID: <DM6PR11MB4491FA1B2F9484E7F30E221D9A1A9@DM6PR11MB4491.namprd11.prod.outlook.com> (raw)
In-Reply-To: <3858a89d-30cd-e63e-ad85-7d1f61fc8564@intel.com>
> >>>>>
> >>>>>> Currently, there is a hard limitation on the PMD power management
> >>>>>> support that only allows it to support a single queue per lcore. This is
> >>>>>> not ideal as most DPDK use cases will poll multiple queues per core.
> >>>>>>
> >>>>>> The PMD power management mechanism relies on ethdev Rx callbacks, so it
> >>>>>> is very difficult to implement such support because callbacks are
> >>>>>> effectively stateless and have no visibility into what the other ethdev
> >>>>>> devices are doing. This places limitations on what we can do within the
> >>>>>> framework of Rx callbacks, but the basics of this implementation are as
> >>>>>> follows:
> >>>>>>
> >>>>>> - Replace per-queue structures with per-lcore ones, so that any device
> >>>>>> polled from the same lcore can share data
> >>>>>> - Any queue that is going to be polled from a specific lcore has to be
> >>>>>> added to the list of queues to poll, so that the callback is aware of
> >>>>>> other queues being polled by the same lcore
> >>>>>> - Both the empty poll counter and the actual power saving mechanism is
> >>>>>> shared between all queues polled on a particular lcore, and is only
> >>>>>> activated when all queues in the list were polled and were determined
> >>>>>> to have no traffic.
> >>>>>> - The limitation on UMWAIT-based polling is not removed because UMWAIT
> >>>>>> is incapable of monitoring more than one address.
> >>>>>>
> >>>>>> Also, while we're at it, update and improve the docs.
> >>>>>>
> >>>>>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> >>>>>> ---
> >>>>>>
> >>>>>> Notes:
> >>>>>> v6:
> >>>>>> - Track each individual queue sleep status (Konstantin)
> >>>>>> - Fix segfault (Dave)
> >>>>>>
> >>>>>> v5:
> >>>>>> - Remove the "power save queue" API and replace it with mechanism suggested by
> >>>>>> Konstantin
> >>>>>>
> >>>>>> v3:
> >>>>>> - Move the list of supported NICs to NIC feature table
> >>>>>>
> >>>>>> v2:
> >>>>>> - Use a TAILQ for queues instead of a static array
> >>>>>> - Address feedback from Konstantin
> >>>>>> - Add additional checks for stopped queues
> >>>>>>
> >>>>
> >>>> <snip>
> >>>>
> >>>>> ....
> >>>>>> +static inline void
> >>>>>> +queue_reset(struct pmd_core_cfg *cfg, struct queue_list_entry *qcfg)
> >>>>>> +{
> >>>>>> + const bool is_ready_to_sleep = qcfg->n_empty_polls > EMPTYPOLL_MAX;
> >>>>>> +
> >>>>>> + /* reset empty poll counter for this queue */
> >>>>>> + qcfg->n_empty_polls = 0;
> >>>>>> + /* reset the queue sleep counter as well */
> >>>>>> + qcfg->n_sleeps = 0;
> >>>>>> + /* remove the queue from list of cores ready to sleep */
> >>>>>> + if (is_ready_to_sleep)
> >>>>>> + cfg->n_queues_ready_to_sleep--;
> >>>>>> + /*
> >>>>>> + * no need change the lcore sleep target counter because this lcore will
> >>>>>> + * reach the n_sleeps anyway, and the other cores are already counted so
> >>>>>> + * there's no need to do anything else.
> >>>>>> + */
> >>>>>> +}
> >>>>>> +
> >>>>>> +static inline bool
> >>>>>> +queue_can_sleep(struct pmd_core_cfg *cfg, struct queue_list_entry *qcfg)
> >>>>>> +{
> >>>>>> + /* this function is called - that means we have an empty poll */
> >>>>>> + qcfg->n_empty_polls++;
> >>>>>> +
> >>>>>> + /* if we haven't reached threshold for empty polls, we can't sleep */
> >>>>>> + if (qcfg->n_empty_polls <= EMPTYPOLL_MAX)
> >>>>>> + return false;
> >>>>>> +
> >>>>>> + /*
> >>>>>> + * we've reached a point where we are able to sleep, but we still need
> >>>>>> + * to check if this queue has already been marked for sleeping.
> >>>>>> + */
> >>>>>> + if (qcfg->n_sleeps == cfg->sleep_target)
> >>>>>> + return true;
> >>>>>> +
> >>>>>> + /* mark this queue as ready for sleep */
> >>>>>> + qcfg->n_sleeps = cfg->sleep_target;
> >>>>>> + cfg->n_queues_ready_to_sleep++;
> >>>>>
> >>>>> So, assuming there is no incoming traffic, should it be:
> >>>>> 1) poll_all_queues(times=EMPTYPOLL_MAX); sleep; poll_all_queues(times=1); sleep; poll_all_queues(times=1); sleep; ...
> >>>>> OR
> >>>>> 2) poll_all_queues(times=EMPTYPOLL_MAX); sleep; poll_all_queues(times= EMPTYPOLL_MAX); sleep; poll_all_queues(times=
> >>>> EMPTYPOLL_MAX); sleep; ...
> >>>>> ?
> >>>>>
> >>>>> My initial thought was 2) but might be the intention is 1)?
> >>>>
> >>>>
> >>>> The intent is 1), not 2). There's no need to wait for more empty polls
> >>>> once we pass the threshold - we keep sleeping until there's traffic.
> >>>>
> >>>
> >>> Ok, then:
> >>> Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> >>>
> >>> Probably worth to put extra explanation here on in the doc,
> >>> to help people avoid wrong assumptions😉
> >>>
> >>
> >> I don't see value in going into such details. What would be the point?
> >> Like, what difference would this information make to anyone?
> >
> > I thought it is obvious: if you put extra explanation into the code,
> > then it would be easier for anyone who reads it (reviewers/maintainers/users)
> > to understand what it supposed to do.
> >
>
> You're suggesting to put this *in the doc*, which implies that *the
> user* will find this information useful. I'm OK with adding this info as
> a comment somewhere perhaps, but why put it in the doc?
I don't really mind where you'll put it, either extra comments in that file,
or few lines in the doc - both are ok to me.
next prev parent reply other threads:[~2021-07-07 17:09 UTC|newest]
Thread overview: 165+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-01 12:00 [dpdk-dev] [PATCH v1 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 1/7] power_intrinsics: allow monitor checks inversion Anatoly Burakov
2021-06-21 12:56 ` Ananyev, Konstantin
2021-06-23 9:43 ` Burakov, Anatoly
2021-06-23 9:55 ` Ananyev, Konstantin
2021-06-23 10:00 ` Burakov, Anatoly
2021-06-23 11:00 ` Ananyev, Konstantin
2021-06-23 12:12 ` Burakov, Anatoly
2021-06-23 13:27 ` Ananyev, Konstantin
2021-06-23 14:13 ` Burakov, Anatoly
2021-06-24 9:47 ` Ananyev, Konstantin
2021-06-24 14:34 ` Burakov, Anatoly
2021-06-24 14:57 ` Ananyev, Konstantin
2021-06-24 15:04 ` Burakov, Anatoly
2021-06-24 15:25 ` Ananyev, Konstantin
2021-06-24 15:54 ` Burakov, Anatoly
2021-07-09 15:03 ` David Marchand
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-02 12:59 ` Loftus, Ciara
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-22 9:13 ` Ananyev, Konstantin
2021-06-23 9:46 ` Burakov, Anatoly
2021-06-23 9:52 ` Ananyev, Konstantin
2021-06-25 11:52 ` Burakov, Anatoly
2021-06-25 14:42 ` Ananyev, Konstantin
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-22 9:41 ` Ananyev, Konstantin
2021-06-23 9:36 ` Burakov, Anatoly
2021-06-23 9:49 ` Ananyev, Konstantin
2021-06-23 9:56 ` Burakov, Anatoly
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 6/7] power: support monitoring " Anatoly Burakov
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-06-28 12:19 ` Ananyev, Konstantin
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-28 12:37 ` Ananyev, Konstantin
2021-06-28 12:43 ` Burakov, Anatoly
2021-06-28 12:58 ` Ananyev, Konstantin
2021-06-28 13:29 ` Burakov, Anatoly
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-28 7:10 ` David Marchand
2021-06-28 9:25 ` Burakov, Anatoly
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 6/7] power: support monitoring " Anatoly Burakov
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-06-28 12:41 ` [dpdk-dev] [PATCH v3 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-28 12:41 ` [dpdk-dev] [PATCH v3 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-06-28 12:41 ` [dpdk-dev] [PATCH v3 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-28 12:41 ` [dpdk-dev] [PATCH v3 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-28 12:41 ` [dpdk-dev] [PATCH v3 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-28 12:41 ` [dpdk-dev] [PATCH v3 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-28 12:41 ` [dpdk-dev] [PATCH v3 6/7] power: support monitoring " Anatoly Burakov
2021-06-28 13:29 ` Ananyev, Konstantin
2021-06-28 14:09 ` Burakov, Anatoly
2021-06-29 0:07 ` Ananyev, Konstantin
2021-06-29 11:05 ` Burakov, Anatoly
2021-06-29 11:39 ` Burakov, Anatoly
2021-06-29 12:14 ` Ananyev, Konstantin
2021-06-29 13:23 ` Burakov, Anatoly
2021-06-28 12:41 ` [dpdk-dev] [PATCH v3 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-06-28 15:54 ` [dpdk-dev] [PATCH v4 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-28 15:54 ` [dpdk-dev] [PATCH v4 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-06-28 15:54 ` [dpdk-dev] [PATCH v4 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-28 15:54 ` [dpdk-dev] [PATCH v4 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-28 15:54 ` [dpdk-dev] [PATCH v4 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-28 15:54 ` [dpdk-dev] [PATCH v4 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-28 15:54 ` [dpdk-dev] [PATCH v4 6/7] power: support monitoring " Anatoly Burakov
2021-06-28 15:54 ` [dpdk-dev] [PATCH v4 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-06-29 15:48 ` [dpdk-dev] [PATCH v5 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-29 15:48 ` [dpdk-dev] [PATCH v5 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-06-29 15:48 ` [dpdk-dev] [PATCH v5 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-29 15:48 ` [dpdk-dev] [PATCH v5 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-29 15:48 ` [dpdk-dev] [PATCH v5 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-29 15:48 ` [dpdk-dev] [PATCH v5 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-30 9:52 ` David Hunt
2021-07-01 9:01 ` David Hunt
2021-07-05 10:24 ` Burakov, Anatoly
2021-06-30 11:04 ` Ananyev, Konstantin
2021-07-05 10:23 ` Burakov, Anatoly
2021-06-29 15:48 ` [dpdk-dev] [PATCH v5 6/7] power: support monitoring " Anatoly Burakov
2021-06-30 10:29 ` Ananyev, Konstantin
2021-07-05 10:08 ` Burakov, Anatoly
2021-06-29 15:48 ` [dpdk-dev] [PATCH v5 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-07-05 15:21 ` [dpdk-dev] [PATCH v6 0/7] Enhancements for PMD power management Anatoly Burakov
2021-07-05 15:21 ` [dpdk-dev] [PATCH v6 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-07-05 15:21 ` [dpdk-dev] [PATCH v6 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-05 15:21 ` [dpdk-dev] [PATCH v6 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-08-04 9:52 ` Kinsella, Ray
2021-07-05 15:21 ` [dpdk-dev] [PATCH v6 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-07 10:14 ` Ananyev, Konstantin
2021-07-05 15:22 ` [dpdk-dev] [PATCH v6 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-06 18:50 ` Ananyev, Konstantin
2021-07-07 10:06 ` Burakov, Anatoly
2021-07-07 10:11 ` Ananyev, Konstantin
2021-07-07 11:54 ` Burakov, Anatoly
2021-07-07 12:51 ` Ananyev, Konstantin
2021-07-07 14:35 ` Burakov, Anatoly
2021-07-07 17:09 ` Ananyev, Konstantin [this message]
2021-07-07 10:04 ` David Hunt
2021-07-07 10:28 ` Burakov, Anatoly
2021-07-05 15:22 ` [dpdk-dev] [PATCH v6 6/7] power: support monitoring " Anatoly Burakov
2021-07-07 10:16 ` Ananyev, Konstantin
2021-07-05 15:22 ` [dpdk-dev] [PATCH v6 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-07-07 10:48 ` [dpdk-dev] [PATCH v7 0/7] Enhancements for PMD power management Anatoly Burakov
2021-07-07 10:48 ` [dpdk-dev] [PATCH v7 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-07-07 11:56 ` David Hunt
2021-07-07 10:48 ` [dpdk-dev] [PATCH v7 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-07 10:48 ` [dpdk-dev] [PATCH v7 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-07-07 12:01 ` David Hunt
2021-07-07 10:48 ` [dpdk-dev] [PATCH v7 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-07 12:02 ` David Hunt
2021-07-07 10:48 ` [dpdk-dev] [PATCH v7 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-07 11:54 ` David Hunt
2021-07-07 10:48 ` [dpdk-dev] [PATCH v7 6/7] power: support monitoring " Anatoly Burakov
2021-07-07 12:03 ` David Hunt
2021-07-07 10:48 ` [dpdk-dev] [PATCH v7 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-07-07 12:03 ` David Hunt
2021-07-08 14:13 ` [dpdk-dev] [PATCH v8 0/7] Enhancements for PMD power management Anatoly Burakov
2021-07-08 14:13 ` [dpdk-dev] [PATCH v8 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-07-08 16:56 ` McDaniel, Timothy
2021-07-09 13:46 ` Thomas Monjalon
2021-07-09 14:41 ` Burakov, Anatoly
2021-07-08 14:13 ` [dpdk-dev] [PATCH v8 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-08 14:13 ` [dpdk-dev] [PATCH v8 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-07-08 14:13 ` [dpdk-dev] [PATCH v8 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-08 14:13 ` [dpdk-dev] [PATCH v8 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-09 14:24 ` David Marchand
2021-07-09 14:42 ` Burakov, Anatoly
2021-07-09 14:46 ` David Marchand
2021-07-09 14:53 ` Burakov, Anatoly
2021-07-08 14:13 ` [dpdk-dev] [PATCH v8 6/7] power: support monitoring " Anatoly Burakov
2021-07-08 14:13 ` [dpdk-dev] [PATCH v8 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-07-09 14:50 ` David Marchand
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 0/8] Enhancements for PMD power management Anatoly Burakov
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 1/8] eal: use callbacks for power monitoring comparison Anatoly Burakov
2021-07-09 16:00 ` Anatoly Burakov
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 2/8] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-09 16:00 ` Anatoly Burakov
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 3/8] doc: add PMD power management NIC feature Anatoly Burakov
2021-07-09 15:57 ` Burakov, Anatoly
2021-07-09 16:00 ` Anatoly Burakov
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 4/8] eal: add power monitor for multiple events Anatoly Burakov
2021-07-09 16:00 ` Anatoly Burakov
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 5/8] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-09 16:00 ` Anatoly Burakov
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 6/8] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-09 16:00 ` Anatoly Burakov
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 7/8] power: support monitoring " Anatoly Burakov
2021-07-09 16:00 ` Anatoly Burakov
2021-07-09 15:53 ` [dpdk-dev] [PATCH v9 8/8] examples/l3fwd-power: support multiq in PMD modes Anatoly Burakov
2021-07-09 16:00 ` Anatoly Burakov
2021-07-09 16:00 ` [dpdk-dev] [PATCH v9 0/8] Enhancements for PMD power management Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 " Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 1/8] eal: use callbacks for power monitoring comparison Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 2/8] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 3/8] doc: add PMD power management NIC feature Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 4/8] eal: add power monitor for multiple events Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 5/8] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 6/8] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 7/8] power: support monitoring " Anatoly Burakov
2021-07-09 16:08 ` [dpdk-dev] [PATCH v10 8/8] examples/l3fwd-power: support multiq in PMD modes Anatoly Burakov
2021-07-09 19:24 ` [dpdk-dev] [PATCH v10 0/8] Enhancements for PMD power management David Marchand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DM6PR11MB4491FA1B2F9484E7F30E221D9A1A9@DM6PR11MB4491.namprd11.prod.outlook.com \
--to=konstantin.ananyev@intel.com \
--cc=anatoly.burakov@intel.com \
--cc=ciara.loftus@intel.com \
--cc=david.hunt@intel.com \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).