DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
	"dev@dpdk.org" <dev@dpdk.org>,
	"Hunt, David" <david.hunt@intel.com>
Cc: "Loftus, Ciara" <ciara.loftus@intel.com>
Subject: Re: [dpdk-dev] [PATCH v6 5/7] power: support callbacks for multiple Rx queues
Date: Wed, 7 Jul 2021 11:06:57 +0100	[thread overview]
Message-ID: <e0b55424-faac-004f-c8b9-e0a9e9dd1bc3@intel.com> (raw)
In-Reply-To: <DM6PR11MB4491E71E4EEA72F0116993E29A1B9@DM6PR11MB4491.namprd11.prod.outlook.com>

On 06-Jul-21 7:50 PM, Ananyev, Konstantin wrote:
> 
>> Currently, there is a hard limitation on the PMD power management
>> support that only allows it to support a single queue per lcore. This is
>> not ideal as most DPDK use cases will poll multiple queues per core.
>>
>> The PMD power management mechanism relies on ethdev Rx callbacks, so it
>> is very difficult to implement such support because callbacks are
>> effectively stateless and have no visibility into what the other ethdev
>> devices are doing. This places limitations on what we can do within the
>> framework of Rx callbacks, but the basics of this implementation are as
>> follows:
>>
>> - Replace per-queue structures with per-lcore ones, so that any device
>>    polled from the same lcore can share data
>> - Any queue that is going to be polled from a specific lcore has to be
>>    added to the list of queues to poll, so that the callback is aware of
>>    other queues being polled by the same lcore
>> - Both the empty poll counter and the actual power saving mechanism is
>>    shared between all queues polled on a particular lcore, and is only
>>    activated when all queues in the list were polled and were determined
>>    to have no traffic.
>> - The limitation on UMWAIT-based polling is not removed because UMWAIT
>>    is incapable of monitoring more than one address.
>>
>> Also, while we're at it, update and improve the docs.
>>
>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
>> ---
>>
>> Notes:
>>      v6:
>>      - Track each individual queue sleep status (Konstantin)
>>      - Fix segfault (Dave)
>>
>>      v5:
>>      - Remove the "power save queue" API and replace it with mechanism suggested by
>>        Konstantin
>>
>>      v3:
>>      - Move the list of supported NICs to NIC feature table
>>
>>      v2:
>>      - Use a TAILQ for queues instead of a static array
>>      - Address feedback from Konstantin
>>      - Add additional checks for stopped queues
>>

<snip>

> ....
>> +static inline void
>> +queue_reset(struct pmd_core_cfg *cfg, struct queue_list_entry *qcfg)
>> +{
>> +     const bool is_ready_to_sleep = qcfg->n_empty_polls > EMPTYPOLL_MAX;
>> +
>> +     /* reset empty poll counter for this queue */
>> +     qcfg->n_empty_polls = 0;
>> +     /* reset the queue sleep counter as well */
>> +     qcfg->n_sleeps = 0;
>> +     /* remove the queue from list of cores ready to sleep */
>> +     if (is_ready_to_sleep)
>> +             cfg->n_queues_ready_to_sleep--;
>> +     /*
>> +      * no need change the lcore sleep target counter because this lcore will
>> +      * reach the n_sleeps anyway, and the other cores are already counted so
>> +      * there's no need to do anything else.
>> +      */
>> +}
>> +
>> +static inline bool
>> +queue_can_sleep(struct pmd_core_cfg *cfg, struct queue_list_entry *qcfg)
>> +{
>> +     /* this function is called - that means we have an empty poll */
>> +     qcfg->n_empty_polls++;
>> +
>> +     /* if we haven't reached threshold for empty polls, we can't sleep */
>> +     if (qcfg->n_empty_polls <= EMPTYPOLL_MAX)
>> +             return false;
>> +
>> +     /*
>> +      * we've reached a point where we are able to sleep, but we still need
>> +      * to check if this queue has already been marked for sleeping.
>> +      */
>> +     if (qcfg->n_sleeps == cfg->sleep_target)
>> +             return true;
>> +
>> +     /* mark this queue as ready for sleep */
>> +     qcfg->n_sleeps = cfg->sleep_target;
>> +     cfg->n_queues_ready_to_sleep++;
> 
> So, assuming there is no incoming traffic, should it be:
> 1) poll_all_queues(times=EMPTYPOLL_MAX); sleep; poll_all_queues(times=1); sleep; poll_all_queues(times=1); sleep; ...
> OR
> 2) poll_all_queues(times=EMPTYPOLL_MAX); sleep; poll_all_queues(times= EMPTYPOLL_MAX); sleep; poll_all_queues(times= EMPTYPOLL_MAX); sleep; ...
> ?
> 
> My initial thought was 2) but might be the intention is 1)?


The intent is 1), not 2). There's no need to wait for more empty polls 
once we pass the threshold - we keep sleeping until there's traffic.


-- 
Thanks,
Anatoly

  reply	other threads:[~2021-07-07 10:07 UTC|newest]

Thread overview: 165+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-01 12:00 [dpdk-dev] [PATCH v1 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 1/7] power_intrinsics: allow monitor checks inversion Anatoly Burakov
2021-06-21 12:56   ` Ananyev, Konstantin
2021-06-23  9:43     ` Burakov, Anatoly
2021-06-23  9:55       ` Ananyev, Konstantin
2021-06-23 10:00         ` Burakov, Anatoly
2021-06-23 11:00           ` Ananyev, Konstantin
2021-06-23 12:12             ` Burakov, Anatoly
2021-06-23 13:27               ` Ananyev, Konstantin
2021-06-23 14:13                 ` Burakov, Anatoly
2021-06-24  9:47                   ` Ananyev, Konstantin
2021-06-24 14:34                     ` Burakov, Anatoly
2021-06-24 14:57                       ` Ananyev, Konstantin
2021-06-24 15:04                         ` Burakov, Anatoly
2021-06-24 15:25                           ` Ananyev, Konstantin
2021-06-24 15:54                             ` Burakov, Anatoly
2021-07-09 15:03                       ` David Marchand
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-02 12:59   ` Loftus, Ciara
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-22  9:13   ` Ananyev, Konstantin
2021-06-23  9:46     ` Burakov, Anatoly
2021-06-23  9:52       ` Ananyev, Konstantin
2021-06-25 11:52         ` Burakov, Anatoly
2021-06-25 14:42           ` Ananyev, Konstantin
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-22  9:41   ` Ananyev, Konstantin
2021-06-23  9:36     ` Burakov, Anatoly
2021-06-23  9:49       ` Ananyev, Konstantin
2021-06-23  9:56         ` Burakov, Anatoly
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 6/7] power: support monitoring " Anatoly Burakov
2021-06-01 12:00 ` [dpdk-dev] [PATCH v1 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-06-25 14:00 ` [dpdk-dev] [PATCH v2 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-25 14:00   ` [dpdk-dev] [PATCH v2 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-06-28 12:19     ` Ananyev, Konstantin
2021-06-25 14:00   ` [dpdk-dev] [PATCH v2 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-25 14:00   ` [dpdk-dev] [PATCH v2 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-28 12:37     ` Ananyev, Konstantin
2021-06-28 12:43       ` Burakov, Anatoly
2021-06-28 12:58         ` Ananyev, Konstantin
2021-06-28 13:29           ` Burakov, Anatoly
2021-06-25 14:00   ` [dpdk-dev] [PATCH v2 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-25 14:00   ` [dpdk-dev] [PATCH v2 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-28  7:10     ` David Marchand
2021-06-28  9:25       ` Burakov, Anatoly
2021-06-25 14:00   ` [dpdk-dev] [PATCH v2 6/7] power: support monitoring " Anatoly Burakov
2021-06-25 14:00   ` [dpdk-dev] [PATCH v2 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-06-28 12:41   ` [dpdk-dev] [PATCH v3 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-28 12:41     ` [dpdk-dev] [PATCH v3 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-06-28 12:41     ` [dpdk-dev] [PATCH v3 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-28 12:41     ` [dpdk-dev] [PATCH v3 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-28 12:41     ` [dpdk-dev] [PATCH v3 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-28 12:41     ` [dpdk-dev] [PATCH v3 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-28 12:41     ` [dpdk-dev] [PATCH v3 6/7] power: support monitoring " Anatoly Burakov
2021-06-28 13:29       ` Ananyev, Konstantin
2021-06-28 14:09         ` Burakov, Anatoly
2021-06-29  0:07           ` Ananyev, Konstantin
2021-06-29 11:05             ` Burakov, Anatoly
2021-06-29 11:39             ` Burakov, Anatoly
2021-06-29 12:14               ` Ananyev, Konstantin
2021-06-29 13:23                 ` Burakov, Anatoly
2021-06-28 12:41     ` [dpdk-dev] [PATCH v3 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-06-28 15:54     ` [dpdk-dev] [PATCH v4 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-28 15:54       ` [dpdk-dev] [PATCH v4 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-06-28 15:54       ` [dpdk-dev] [PATCH v4 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-28 15:54       ` [dpdk-dev] [PATCH v4 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-28 15:54       ` [dpdk-dev] [PATCH v4 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-28 15:54       ` [dpdk-dev] [PATCH v4 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-28 15:54       ` [dpdk-dev] [PATCH v4 6/7] power: support monitoring " Anatoly Burakov
2021-06-28 15:54       ` [dpdk-dev] [PATCH v4 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-06-29 15:48       ` [dpdk-dev] [PATCH v5 0/7] Enhancements for PMD power management Anatoly Burakov
2021-06-29 15:48         ` [dpdk-dev] [PATCH v5 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-06-29 15:48         ` [dpdk-dev] [PATCH v5 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-06-29 15:48         ` [dpdk-dev] [PATCH v5 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-06-29 15:48         ` [dpdk-dev] [PATCH v5 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-06-29 15:48         ` [dpdk-dev] [PATCH v5 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-06-30  9:52           ` David Hunt
2021-07-01  9:01             ` David Hunt
2021-07-05 10:24               ` Burakov, Anatoly
2021-06-30 11:04           ` Ananyev, Konstantin
2021-07-05 10:23             ` Burakov, Anatoly
2021-06-29 15:48         ` [dpdk-dev] [PATCH v5 6/7] power: support monitoring " Anatoly Burakov
2021-06-30 10:29           ` Ananyev, Konstantin
2021-07-05 10:08             ` Burakov, Anatoly
2021-06-29 15:48         ` [dpdk-dev] [PATCH v5 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-07-05 15:21         ` [dpdk-dev] [PATCH v6 0/7] Enhancements for PMD power management Anatoly Burakov
2021-07-05 15:21           ` [dpdk-dev] [PATCH v6 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-07-05 15:21           ` [dpdk-dev] [PATCH v6 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-05 15:21           ` [dpdk-dev] [PATCH v6 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-08-04  9:52             ` Kinsella, Ray
2021-07-05 15:21           ` [dpdk-dev] [PATCH v6 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-07 10:14             ` Ananyev, Konstantin
2021-07-05 15:22           ` [dpdk-dev] [PATCH v6 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-06 18:50             ` Ananyev, Konstantin
2021-07-07 10:06               ` Burakov, Anatoly [this message]
2021-07-07 10:11                 ` Ananyev, Konstantin
2021-07-07 11:54                   ` Burakov, Anatoly
2021-07-07 12:51                     ` Ananyev, Konstantin
2021-07-07 14:35                       ` Burakov, Anatoly
2021-07-07 17:09                         ` Ananyev, Konstantin
2021-07-07 10:04             ` David Hunt
2021-07-07 10:28               ` Burakov, Anatoly
2021-07-05 15:22           ` [dpdk-dev] [PATCH v6 6/7] power: support monitoring " Anatoly Burakov
2021-07-07 10:16             ` Ananyev, Konstantin
2021-07-05 15:22           ` [dpdk-dev] [PATCH v6 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-07-07 10:48           ` [dpdk-dev] [PATCH v7 0/7] Enhancements for PMD power management Anatoly Burakov
2021-07-07 10:48             ` [dpdk-dev] [PATCH v7 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-07-07 11:56               ` David Hunt
2021-07-07 10:48             ` [dpdk-dev] [PATCH v7 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-07 10:48             ` [dpdk-dev] [PATCH v7 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-07-07 12:01               ` David Hunt
2021-07-07 10:48             ` [dpdk-dev] [PATCH v7 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-07 12:02               ` David Hunt
2021-07-07 10:48             ` [dpdk-dev] [PATCH v7 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-07 11:54               ` David Hunt
2021-07-07 10:48             ` [dpdk-dev] [PATCH v7 6/7] power: support monitoring " Anatoly Burakov
2021-07-07 12:03               ` David Hunt
2021-07-07 10:48             ` [dpdk-dev] [PATCH v7 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-07-07 12:03               ` David Hunt
2021-07-08 14:13             ` [dpdk-dev] [PATCH v8 0/7] Enhancements for PMD power management Anatoly Burakov
2021-07-08 14:13               ` [dpdk-dev] [PATCH v8 1/7] power_intrinsics: use callbacks for comparison Anatoly Burakov
2021-07-08 16:56                 ` McDaniel, Timothy
2021-07-09 13:46                 ` Thomas Monjalon
2021-07-09 14:41                   ` Burakov, Anatoly
2021-07-08 14:13               ` [dpdk-dev] [PATCH v8 2/7] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-08 14:13               ` [dpdk-dev] [PATCH v8 3/7] eal: add power monitor for multiple events Anatoly Burakov
2021-07-08 14:13               ` [dpdk-dev] [PATCH v8 4/7] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-08 14:13               ` [dpdk-dev] [PATCH v8 5/7] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-09 14:24                 ` David Marchand
2021-07-09 14:42                   ` Burakov, Anatoly
2021-07-09 14:46                     ` David Marchand
2021-07-09 14:53                       ` Burakov, Anatoly
2021-07-08 14:13               ` [dpdk-dev] [PATCH v8 6/7] power: support monitoring " Anatoly Burakov
2021-07-08 14:13               ` [dpdk-dev] [PATCH v8 7/7] l3fwd-power: support multiqueue in PMD pmgmt modes Anatoly Burakov
2021-07-09 14:50                 ` David Marchand
2021-07-09 15:53               ` [dpdk-dev] [PATCH v9 0/8] Enhancements for PMD power management Anatoly Burakov
2021-07-09 15:53                 ` [dpdk-dev] [PATCH v9 1/8] eal: use callbacks for power monitoring comparison Anatoly Burakov
2021-07-09 16:00                   ` Anatoly Burakov
2021-07-09 15:53                 ` [dpdk-dev] [PATCH v9 2/8] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-09 16:00                   ` Anatoly Burakov
2021-07-09 15:53                 ` [dpdk-dev] [PATCH v9 3/8] doc: add PMD power management NIC feature Anatoly Burakov
2021-07-09 15:57                   ` Burakov, Anatoly
2021-07-09 16:00                   ` Anatoly Burakov
2021-07-09 15:53                 ` [dpdk-dev] [PATCH v9 4/8] eal: add power monitor for multiple events Anatoly Burakov
2021-07-09 16:00                   ` Anatoly Burakov
2021-07-09 15:53                 ` [dpdk-dev] [PATCH v9 5/8] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-09 16:00                   ` Anatoly Burakov
2021-07-09 15:53                 ` [dpdk-dev] [PATCH v9 6/8] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-09 16:00                   ` Anatoly Burakov
2021-07-09 15:53                 ` [dpdk-dev] [PATCH v9 7/8] power: support monitoring " Anatoly Burakov
2021-07-09 16:00                   ` Anatoly Burakov
2021-07-09 15:53                 ` [dpdk-dev] [PATCH v9 8/8] examples/l3fwd-power: support multiq in PMD modes Anatoly Burakov
2021-07-09 16:00                   ` Anatoly Burakov
2021-07-09 16:00                 ` [dpdk-dev] [PATCH v9 0/8] Enhancements for PMD power management Anatoly Burakov
2021-07-09 16:08                 ` [dpdk-dev] [PATCH v10 " Anatoly Burakov
2021-07-09 16:08                   ` [dpdk-dev] [PATCH v10 1/8] eal: use callbacks for power monitoring comparison Anatoly Burakov
2021-07-09 16:08                   ` [dpdk-dev] [PATCH v10 2/8] net/af_xdp: add power monitor support Anatoly Burakov
2021-07-09 16:08                   ` [dpdk-dev] [PATCH v10 3/8] doc: add PMD power management NIC feature Anatoly Burakov
2021-07-09 16:08                   ` [dpdk-dev] [PATCH v10 4/8] eal: add power monitor for multiple events Anatoly Burakov
2021-07-09 16:08                   ` [dpdk-dev] [PATCH v10 5/8] power: remove thread safety from PMD power API's Anatoly Burakov
2021-07-09 16:08                   ` [dpdk-dev] [PATCH v10 6/8] power: support callbacks for multiple Rx queues Anatoly Burakov
2021-07-09 16:08                   ` [dpdk-dev] [PATCH v10 7/8] power: support monitoring " Anatoly Burakov
2021-07-09 16:08                   ` [dpdk-dev] [PATCH v10 8/8] examples/l3fwd-power: support multiq in PMD modes Anatoly Burakov
2021-07-09 19:24                   ` [dpdk-dev] [PATCH v10 0/8] Enhancements for PMD power management David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e0b55424-faac-004f-c8b9-e0a9e9dd1bc3@intel.com \
    --to=anatoly.burakov@intel.com \
    --cc=ciara.loftus@intel.com \
    --cc=david.hunt@intel.com \
    --cc=dev@dpdk.org \
    --cc=konstantin.ananyev@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).