DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
	"dev@dpdk.org" <dev@dpdk.org>
Cc: "Ma, Liang J" <liang.j.ma@intel.com>,
	"Hunt, David" <david.hunt@intel.com>,
	Ray Kinsella <mdr@ashroe.eu>, Neil Horman <nhorman@tuxdriver.com>,
	"thomas@monjalon.net" <thomas@monjalon.net>,
	"McDaniel, Timothy" <timothy.mcdaniel@intel.com>,
	"Richardson, Bruce" <bruce.richardson@intel.com>,
	"Macnamara, Chris" <chris.macnamara@intel.com>
Subject: Re: [dpdk-dev] [PATCH v16 07/11] power: add PMD power management API and callback
Date: Thu, 14 Jan 2021 13:00:07 +0000	[thread overview]
Message-ID: <d981e96f-b792-9db2-f0c0-ab0c0b997b00@intel.com> (raw)
In-Reply-To: <5044aa4a-ed24-df3e-b0b2-cf6e05d839a8@intel.com>

On 13-Jan-21 5:29 PM, Burakov, Anatoly wrote:
> On 13-Jan-21 12:58 PM, Ananyev, Konstantin wrote:
>>
>>
>>> -----Original Message-----
>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>> Sent: Tuesday, January 12, 2021 5:37 PM
>>> To: dev@dpdk.org
>>> Cc: Ma, Liang J <liang.j.ma@intel.com>; Hunt, David 
>>> <david.hunt@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
>>> <nhorman@tuxdriver.com>; thomas@monjalon.net; Ananyev, Konstantin 
>>> <konstantin.ananyev@intel.com>; McDaniel, Timothy
>>> <timothy.mcdaniel@intel.com>; Richardson, Bruce 
>>> <bruce.richardson@intel.com>; Macnamara, Chris 
>>> <chris.macnamara@intel.com>
>>> Subject: [PATCH v16 07/11] power: add PMD power management API and 
>>> callback
>>>
>>> From: Liang Ma <liang.j.ma@intel.com>
>>>
>>> Add a simple on/off switch that will enable saving power when no
>>> packets are arriving. It is based on counting the number of empty
>>> polls and, when the number reaches a certain threshold, entering an
>>> architecture-defined optimized power state that will either wait
>>> until a TSC timestamp expires, or when packets arrive.
>>>
>>> This API mandates a core-to-single-queue mapping (that is, multiple
>>> queued per device are supported, but they have to be polled on different
>>> cores).
>>>
>>> This design is using PMD RX callbacks.
>>>
>>> 1. UMWAIT/UMONITOR:
>>>
>>>     When a certain threshold of empty polls is reached, the core will go
>>>     into a power optimized sleep while waiting on an address of next RX
>>>     descriptor to be written to.
>>>
>>> 2. TPAUSE/Pause instruction
>>>
>>>     This method uses the pause (or TPAUSE, if available) instruction to
>>>     avoid busy polling.
>>>
>>> 3. Frequency scaling
>>>     Reuse existing DPDK power library to scale up/down core frequency
>>>     depending on traffic volume.
>>>
>>> Signed-off-by: Liang Ma <liang.j.ma@intel.com>
>>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>> ---
>>>
>>> Notes:
>>>      v15:
>>>      - Fix check in UMWAIT callback
>>>
>>>      v13:
>>>      - Rework the synchronization mechanism to not require locking
>>>      - Add more parameter checking
>>>      - Rework n_rx_queues access to not go through internal PMD 
>>> structures and use
>>>        public API instead
>>>
>>>      v13:
>>>      - Rework the synchronization mechanism to not require locking
>>>      - Add more parameter checking
>>>      - Rework n_rx_queues access to not go through internal PMD 
>>> structures and use
>>>        public API instead
>>>
>>>   doc/guides/prog_guide/power_man.rst    |  44 +++
>>>   doc/guides/rel_notes/release_21_02.rst |  10 +
>>>   lib/librte_power/meson.build           |   5 +-
>>>   lib/librte_power/rte_power_pmd_mgmt.c  | 359 +++++++++++++++++++++++++
>>>   lib/librte_power/rte_power_pmd_mgmt.h  |  90 +++++++
>>>   lib/librte_power/version.map           |   5 +
>>>   6 files changed, 511 insertions(+), 2 deletions(-)
>>>   create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c
>>>   create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
>>>
>>
>> ...
>>
>>> +
>>> +static uint16_t
>>> +clb_umwait(uint16_t port_id, uint16_t qidx, struct rte_mbuf **pkts 
>>> __rte_unused,
>>> +uint16_t nb_rx, uint16_t max_pkts __rte_unused,
>>> +void *addr __rte_unused)
>>> +{
>>> +
>>> +struct pmd_queue_cfg *q_conf;
>>> +
>>> +q_conf = &port_cfg[port_id][qidx];
>>> +
>>> +if (unlikely(nb_rx == 0)) {
>>> +q_conf->empty_poll_stats++;
>>> +if (unlikely(q_conf->empty_poll_stats > EMPTYPOLL_MAX)) {
>>> +struct rte_power_monitor_cond pmc;
>>> +uint16_t ret;
>>> +
>>> +/*
>>> + * we might get a cancellation request while being
>>> + * inside the callback, in which case the wakeup
>>> + * wouldn't work because it would've arrived too early.
>>> + *
>>> + * to get around this, we notify the other thread that
>>> + * we're sleeping, so that it can spin until we're done.
>>> + * unsolicited wakeups are perfectly safe.
>>> + */
>>> +q_conf->umwait_in_progress = true;
>>
>> This write and subsequent read can be reordered by the cpu.
>> I think you need rte_atomic_thread_fence(__ATOMIC_SEQ_CST) here and
>> in disable() code-path below.
>>
>>> +
>>> +/* check if we need to cancel sleep */
>>> +if (q_conf->pwr_mgmt_state == PMD_MGMT_ENABLED) {
>>> +/* use monitoring condition to sleep */
>>> +ret = rte_eth_get_monitor_addr(port_id, qidx,
>>> +&pmc);
>>> +if (ret == 0)
>>> +rte_power_monitor(&pmc, -1ULL);
>>> +}
>>> +q_conf->umwait_in_progress = false;
>>> +}
>>> +} else
>>> +q_conf->empty_poll_stats = 0;
>>> +
>>> +return nb_rx;
>>> +}
>>> +
>>
>> ...
>>
>>> +
>>> +int
>>> +rte_power_pmd_mgmt_queue_disable(unsigned int lcore_id,
>>> +uint16_t port_id, uint16_t queue_id)
>>> +{
>>> +struct pmd_queue_cfg *queue_cfg;
>>> +
>>> +RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
>>> +
>>> +if (lcore_id >= RTE_MAX_LCORE || queue_id >= RTE_MAX_QUEUES_PER_PORT)
>>> +return -EINVAL;
>>> +
>>> +/* no need to check queue id as wrong queue id would not be enabled */
>>> +queue_cfg = &port_cfg[port_id][queue_id];
>>> +
>>> +if (queue_cfg->pwr_mgmt_state != PMD_MGMT_ENABLED)
>>> +return -EINVAL;
>>> +
>>> +/* let the callback know we're shutting down */
>>> +queue_cfg->pwr_mgmt_state = PMD_MGMT_BUSY;
>>
>> Same as above - write to pwr_mgmt_state and read from umwait_in_progress
>> could be reordered by cpu.
>> Need to insert rte_atomic_thread_fence(__ATOMIC_SEQ_CST) between them.
>>
>> BTW, out of curiosity - why do you need this intermediate
>> state (PMD_MGMT_BUSY) at all?
>> Why not directly:
>> queue_cfg->pwr_mgmt_state = PMD_MGMT_DISABLED;
>> ?
>>
> 
> Thanks for suggestions, i'll add those.
> 
> The goal for the "intermediate" step is to prevent Rx callback from 
> sleeping in the first place. We can't "wake up" earlier than it goes to 
> sleep, but we may get a request to disable power management while we're 
> at the beginning of the callback and haven't yet entered the 
> rte_power_monitor code.
> 
> In this case, setting it to "BUSY" will prevent the callback from ever 
> sleeping in the first place (see rte_power_pmd_mgmt:108 check), and will 
> unset the "umwait in progress" if there was any.
> 
> So, we have three situations to handle:
> 
> 1) wake up during umwait
> 2) "wake up" during callback after we've set the "umwait in progress" 
> flag but before actual umwait happens - we don't wait to exit before 
> we're sure there's nothing sleeping there
> 3) "wake up" during callback before we set the "umwait in progress" flag
> 
> 1) is handled by the rte_power_monitor_wakeup() call, so that's taken 
> care of. 2) is handled by the other thread waiting on "umwait in 
> progress" becoming false. 3) is handled by having this BUSY check in the 
> umwait thread.
> 
> Hope that made sense!
> 

On further thoughts, the "BUSY" thing relies on a hidden assumption that 
enable/disable power management per queue is supposed to be thread safe. 
If we let go of this assumption, we can get by with just enable/disable, 
so i think i'll just document the thread safety and leave out the "BUSY" 
part.

-- 
Thanks,
Anatoly

  reply	other threads:[~2021-01-14 13:00 UTC|newest]

Thread overview: 421+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-27 17:02 [dpdk-dev] [RFC 0/6] Power-optimized RX for Ethernet devices Anatoly Burakov
2020-05-27 17:02 ` [dpdk-dev] [RFC 1/6] eal: add power management intrinsics Anatoly Burakov
2020-05-28 11:39   ` Ananyev, Konstantin
2020-05-28 14:40     ` Burakov, Anatoly
2020-05-28 14:58       ` Bruce Richardson
2020-05-28 15:38       ` Ananyev, Konstantin
2020-05-29  6:56         ` Jerin Jacob
2020-06-02 10:15           ` Ananyev, Konstantin
2020-06-03  6:22           ` Honnappa Nagarahalli
2020-06-03  6:31             ` Jerin Jacob
2020-11-02 11:09   ` [dpdk-dev] [PATCH v11 0/6] Add PMD power mgmt Liang Ma
2020-11-02 11:10     ` [dpdk-dev] [PATCH v11 1/6] ethdev: add simple power management API Liang Ma
2020-11-02 12:23       ` Burakov, Anatoly
2020-11-02 11:10     ` [dpdk-dev] [PATCH v11 2/6] power: add PMD power management API and callback Liang Ma
2020-11-02 11:10     ` [dpdk-dev] [PATCH v11 3/6] net/ixgbe: implement power management API Liang Ma
2020-11-02 11:10     ` [dpdk-dev] [PATCH v11 4/6] net/i40e: " Liang Ma
2020-11-02 11:10     ` [dpdk-dev] [PATCH v11 5/6] net/ice: " Liang Ma
2020-11-02 11:10     ` [dpdk-dev] [PATCH v11 6/6] examples/l3fwd-power: enable PMD power mgmt Liang Ma
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 00/11] Add PMD power management Anatoly Burakov
2020-12-17 16:12       ` David Marchand
2021-01-08 16:42         ` Burakov, Anatoly
2021-01-11  8:44           ` David Marchand
2021-01-11  8:52             ` David Marchand
2021-01-11 10:21               ` Burakov, Anatoly
2021-01-08 17:42       ` [dpdk-dev] [PATCH v13 " Anatoly Burakov
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 01/11] eal: uninline power intrinsics Anatoly Burakov
2021-01-12 15:54           ` Ananyev, Konstantin
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 02/11] eal: avoid invalid API usage in " Anatoly Burakov
2021-01-08 19:58           ` Stephen Hemminger
2021-01-11 10:21             ` Burakov, Anatoly
2021-01-12 15:56           ` Ananyev, Konstantin
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 03/11] eal: change API of " Anatoly Burakov
2021-01-12 15:58           ` Ananyev, Konstantin
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 04/11] eal: remove sync version of power monitor Anatoly Burakov
2021-01-12 15:59           ` Ananyev, Konstantin
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 05/11] eal: add monitor wakeup function Anatoly Burakov
2021-01-12 16:02           ` Ananyev, Konstantin
2021-01-12 16:18             ` Burakov, Anatoly
2021-01-12 16:25               ` Burakov, Anatoly
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 06/11] ethdev: add simple power management API Anatoly Burakov
2021-01-09  8:04           ` Andrew Rybchenko
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 07/11] power: add PMD power management API and callback Anatoly Burakov
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 08/11] net/ixgbe: implement power management API Anatoly Burakov
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 09/11] net/i40e: " Anatoly Burakov
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 10/11] net/ice: " Anatoly Burakov
2021-01-08 17:42         ` [dpdk-dev] [PATCH v13 11/11] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2021-01-11 14:35         ` [dpdk-dev] [PATCH v14 00/11] Add PMD power management Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 01/11] eal: uninline power intrinsics Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 02/11] eal: avoid invalid API usage in " Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 03/11] eal: change API of " Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 04/11] eal: remove sync version of power monitor Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 05/11] eal: add monitor wakeup function Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 06/11] ethdev: add simple power management API Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 07/11] power: add PMD power management API and callback Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 08/11] net/ixgbe: implement power management API Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 09/11] net/i40e: " Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 10/11] net/ice: " Anatoly Burakov
2021-01-11 14:35           ` [dpdk-dev] [PATCH v14 11/11] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2021-01-11 14:58           ` [dpdk-dev] [PATCH v15 00/11] Add PMD power management Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 01/11] eal: uninline power intrinsics Anatoly Burakov
2021-01-12 16:09               ` Ananyev, Konstantin
2021-01-12 16:14                 ` Burakov, Anatoly
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 02/11] eal: avoid invalid API usage in " Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 03/11] eal: change API of " Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 04/11] eal: remove sync version of power monitor Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 05/11] eal: add monitor wakeup function Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 06/11] ethdev: add simple power management API Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 07/11] power: add PMD power management API and callback Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 08/11] net/ixgbe: implement power management API Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 09/11] net/i40e: " Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 10/11] net/ice: " Anatoly Burakov
2021-01-11 14:58             ` [dpdk-dev] [PATCH v15 11/11] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2021-01-12 17:37             ` [dpdk-dev] [PATCH v16 00/11] Add PMD power management Anatoly Burakov
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 01/11] eal: uninline power intrinsics Anatoly Burakov
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 02/11] eal: avoid invalid API usage in " Anatoly Burakov
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 03/11] eal: change API of " Anatoly Burakov
2021-01-13 13:01                 ` Ananyev, Konstantin
2021-01-13 17:22                   ` Burakov, Anatoly
2021-01-13 18:01                     ` Ananyev, Konstantin
2021-01-14 10:23                       ` Burakov, Anatoly
2021-01-14 12:33                         ` Ananyev, Konstantin
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 04/11] eal: remove sync version of power monitor Anatoly Burakov
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 05/11] eal: add monitor wakeup function Anatoly Burakov
2021-01-13 12:46                 ` Ananyev, Konstantin
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 06/11] ethdev: add simple power management API Anatoly Burakov
2021-01-13 13:18                 ` Ananyev, Konstantin
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 07/11] power: add PMD power management API and callback Anatoly Burakov
2021-01-13 12:58                 ` Ananyev, Konstantin
2021-01-13 17:29                   ` Burakov, Anatoly
2021-01-14 13:00                     ` Burakov, Anatoly [this message]
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 08/11] net/ixgbe: implement power management API Anatoly Burakov
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 09/11] net/i40e: " Anatoly Burakov
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 10/11] net/ice: " Anatoly Burakov
2021-01-12 17:37               ` [dpdk-dev] [PATCH v16 11/11] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2021-01-14  9:36               ` [dpdk-dev] [PATCH v16 00/11] Add PMD power management David Marchand
2021-01-14 10:25                 ` Burakov, Anatoly
2021-01-14 14:46               ` [dpdk-dev] [PATCH v17 " Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 01/11] eal: uninline power intrinsics Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 02/11] eal: avoid invalid API usage in " Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 03/11] eal: change API of " Anatoly Burakov
2021-01-18 22:26                   ` Thomas Monjalon
2021-01-19 10:29                     ` Burakov, Anatoly
2021-01-19 10:42                       ` Thomas Monjalon
2021-01-19 11:23                         ` Burakov, Anatoly
2021-01-19 14:17                           ` Thomas Monjalon
2021-01-20 10:32                             ` Burakov, Anatoly
2021-01-20 10:38                               ` Thomas Monjalon
2021-01-20 11:05                                 ` Burakov, Anatoly
2021-01-20 11:11                                   ` Thomas Monjalon
2021-01-20 11:17                                     ` Burakov, Anatoly
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 04/11] eal: remove sync version of power monitor Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 05/11] eal: add monitor wakeup function Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 06/11] ethdev: add simple power management API Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 07/11] power: add PMD power management API and callback Anatoly Burakov
2021-01-18 12:41                   ` David Hunt
2021-01-19 16:45                     ` [dpdk-dev] [PATCH v18 0/2] Add PMD power management Anatoly Burakov
2021-01-19 16:45                       ` [dpdk-dev] [PATCH v18 1/2] power: add PMD power management API and callback Anatoly Burakov
2021-01-19 16:45                       ` [dpdk-dev] [PATCH v18 2/2] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2021-01-20 11:50                       ` [dpdk-dev] [PATCH v19 0/4] Add PMD power management Anatoly Burakov
2021-01-20 11:50                         ` [dpdk-dev] [PATCH v19 1/4] eal: rename power monitor condition member Anatoly Burakov
2021-01-20 11:50                         ` [dpdk-dev] [PATCH v19 2/4] eal: improve comments around power monitoring API Anatoly Burakov
2021-01-20 11:50                         ` [dpdk-dev] [PATCH v19 3/4] power: add PMD power management API and callback Anatoly Burakov
2021-01-20 11:50                         ` [dpdk-dev] [PATCH v19 4/4] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2021-01-22 17:12                         ` [dpdk-dev] [PATCH v20 0/4] Add PMD power management Anatoly Burakov
2021-01-22 17:12                           ` [dpdk-dev] [PATCH v20 1/4] eal: rename power monitor condition member Anatoly Burakov
2021-01-29 11:26                             ` Thomas Monjalon
2021-01-22 17:12                           ` [dpdk-dev] [PATCH v20 2/4] eal: improve comments around power monitoring API Anatoly Burakov
2021-01-29 11:27                             ` Thomas Monjalon
2021-01-22 17:12                           ` [dpdk-dev] [PATCH v20 3/4] power: add PMD power management API and callback Anatoly Burakov
2021-01-22 17:12                           ` [dpdk-dev] [PATCH v20 4/4] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2021-01-29 14:15                             ` Thomas Monjalon
2021-01-29 14:20                           ` [dpdk-dev] [PATCH v20 0/4] Add PMD power management Thomas Monjalon
2021-01-29 14:47                             ` Burakov, Anatoly
2021-01-18 22:48                   ` [dpdk-dev] [PATCH v17 07/11] power: add PMD power management API and callback Thomas Monjalon
2021-01-19 12:25                     ` Burakov, Anatoly
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 08/11] net/ixgbe: implement power management API Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 09/11] net/i40e: " Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 10/11] net/ice: " Anatoly Burakov
2021-01-14 14:46                 ` [dpdk-dev] [PATCH v17 11/11] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2021-01-18 12:53                   ` David Hunt
2021-01-18 15:24                 ` [dpdk-dev] [PATCH v17 00/11] Add PMD power management David Marchand
2021-01-18 15:45                   ` Burakov, Anatoly
2021-01-18 16:06                     ` Thomas Monjalon
2021-01-18 17:02                       ` Burakov, Anatoly
2021-01-18 17:54                         ` David Marchand
2021-01-18 22:52                 ` Thomas Monjalon
2021-01-19 10:30                   ` Burakov, Anatoly
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 01/11] eal: uninline power intrinsics Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 02/11] eal: avoid invalid API usage in " Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 03/11] eal: change API of " Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 04/11] eal: remove sync version of power monitor Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 05/11] eal: add monitor wakeup function Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 06/11] ethdev: add simple power management API Anatoly Burakov
2020-12-28 11:00       ` Andrew Rybchenko
2021-01-08 16:30         ` Burakov, Anatoly
2021-01-12 20:32       ` Lance Richardson
2021-01-13 13:04         ` Burakov, Anatoly
2021-01-13 13:25           ` Ananyev, Konstantin
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 07/11] power: add PMD power management API and callback Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 08/11] net/ixgbe: implement power management API Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 09/11] net/i40e: " Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 10/11] net/ice: " Anatoly Burakov
2020-12-17 14:05     ` [dpdk-dev] [PATCH v12 11/11] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2020-05-27 17:02 ` [dpdk-dev] [RFC 2/6] ethdev: add simple power management API Anatoly Burakov
2020-05-28 12:15   ` Ananyev, Konstantin
2020-05-27 17:02 ` [dpdk-dev] [RFC 3/6] net/ixgbe: implement " Anatoly Burakov
2020-05-27 17:02 ` [dpdk-dev] [RFC 4/6] net/i40e: " Anatoly Burakov
2020-05-27 17:02 ` [dpdk-dev] [RFC 5/6] net/ice: " Anatoly Burakov
2020-05-27 17:02 ` [dpdk-dev] [RFC 6/6] app/testpmd: add command for power management on a port Anatoly Burakov
2020-05-27 17:33 ` [dpdk-dev] [RFC 0/6] Power-optimized RX for Ethernet devices Jerin Jacob
2020-05-27 20:57   ` Stephen Hemminger
2020-08-11 10:27 ` [dpdk-dev] [RFC v2 1/5] eal: add power management intrinsics Liang Ma
2020-08-11 10:27   ` [dpdk-dev] [RFC v2 2/5] ethdev: add simple power management API and callback Liang Ma
2020-08-13 18:11     ` Liang, Ma
2020-08-11 10:27   ` [dpdk-dev] [RFC v2 3/5] net/ixgbe: implement power management API Liang Ma
2020-08-11 10:27   ` [dpdk-dev] [RFC v2 4/5] net/i40e: " Liang Ma
2020-08-11 10:27   ` [dpdk-dev] [RFC v2 5/5] net/ice: " Liang Ma
2020-08-13 18:04   ` [dpdk-dev] [RFC v2 1/5] eal: add power management intrinsics Liang, Ma
2020-09-03 16:06   ` [dpdk-dev] [RFC PATCH v3 1/6] " Liang Ma
2020-09-03 16:07     ` [dpdk-dev] [RFC PATCH v3 2/6] ethdev: add simple power management API Liang Ma
2020-09-03 16:07     ` [dpdk-dev] [RFC PATCH v3 3/6] power: add simple power management API and callback Liang Ma
2020-09-03 16:07     ` [dpdk-dev] [RFC PATCH v3 4/6] net/ixgbe: implement power management API Liang Ma
2020-09-03 16:07     ` [dpdk-dev] [RFC PATCH v3 5/6] net/i40e: " Liang Ma
2020-09-03 16:07     ` [dpdk-dev] [RFC PATCH v3 6/6] net/ice: " Liang Ma
2020-09-04 10:18   ` [dpdk-dev] [PATCH v3 1/6] eal: add power management intrinsics Liang Ma
2020-09-04 10:18     ` [dpdk-dev] [PATCH v3 2/6] ethdev: add simple power management API Liang Ma
2020-09-04 16:37       ` Stephen Hemminger
2020-09-14 21:04         ` Liang, Ma
2020-09-04 20:54       ` Ananyev, Konstantin
2020-09-04 10:18     ` [dpdk-dev] [PATCH v3 3/6] power: add simple power management API and callback Liang Ma
2020-09-04 16:36       ` Stephen Hemminger
2020-09-14 20:52         ` Liang, Ma
2020-09-04 18:33       ` Ananyev, Konstantin
2020-09-14 21:01         ` Liang, Ma
2020-09-16 14:53           ` Ananyev, Konstantin
2020-09-16 16:39             ` Liang, Ma
2020-09-16 16:44               ` Ananyev, Konstantin
2020-09-04 10:18     ` [dpdk-dev] [PATCH v3 4/6] net/ixgbe: implement power management API Liang Ma
2020-09-04 10:18     ` [dpdk-dev] [PATCH v3 5/6] net/i40e: " Liang Ma
2020-09-04 10:19     ` [dpdk-dev] [PATCH v3 6/6] net/ice: " Liang Ma
2020-09-04 16:23     ` [dpdk-dev] [PATCH v3 1/6] eal: add power management intrinsics Stephen Hemminger
2020-09-14 20:48       ` Liang, Ma
2020-09-04 16:37     ` Stephen Hemminger
2020-09-14 20:49       ` Liang, Ma
2020-09-04 18:42     ` Stephen Hemminger
2020-09-14 21:12       ` Liang, Ma
2020-09-16 16:34       ` Liang, Ma
2020-09-06 21:44     ` Ananyev, Konstantin
2020-09-18  5:01     ` Jerin Jacob
2020-10-02 14:11     ` [dpdk-dev] [PATCH v4 01/10] eal: add new x86 cpuid support for WAITPKG Liang Ma
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 02/10] eal: add power management intrinsics Liang Ma
2020-10-08  8:33         ` Thomas Monjalon
2020-10-08  8:44           ` Jerin Jacob
2020-10-08  9:41             ` Thomas Monjalon
2020-10-08 13:26             ` Burakov, Anatoly
2020-10-08 15:13               ` Jerin Jacob
2020-10-08 17:07                 ` Ananyev, Konstantin
2020-10-09  5:42                   ` Jerin Jacob
2020-10-09  9:25                     ` Burakov, Anatoly
2020-10-09  9:29                       ` Thomas Monjalon
2020-10-09  9:40                         ` Burakov, Anatoly
2020-10-09  9:54                           ` Jerin Jacob
2020-10-09 10:03                             ` Burakov, Anatoly
2020-10-09 10:17                               ` Thomas Monjalon
2020-10-09 10:22                                 ` Burakov, Anatoly
2020-10-09 10:45                                   ` Jerin Jacob
2020-10-09 10:48                                   ` Ananyev, Konstantin
2020-10-09 11:12                                     ` Burakov, Anatoly
2020-10-09 11:36                                       ` Bruce Richardson
2020-10-09 11:42                                         ` Burakov, Anatoly
2020-10-09 10:19                               ` Jerin Jacob
2020-10-08 17:15         ` Ananyev, Konstantin
2020-10-09  9:11           ` Burakov, Anatoly
2020-10-09 15:39             ` Ananyev, Konstantin
2020-10-09 16:10               ` Burakov, Anatoly
2020-10-09 16:56                 ` Ananyev, Konstantin
2020-10-09 16:59                   ` Burakov, Anatoly
2020-10-10 13:19                     ` Ananyev, Konstantin
2020-10-12 10:35                       ` Burakov, Anatoly
2020-10-12 10:36                         ` Burakov, Anatoly
2020-10-12 12:50                           ` Ananyev, Konstantin
2020-10-12 13:13                             ` Burakov, Anatoly
2020-10-13  9:45                               ` Burakov, Anatoly
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 03/10] ethdev: add simple power management API Liang Ma
2020-10-08  8:46         ` Thomas Monjalon
2020-10-08 11:39           ` Ananyev, Konstantin
2020-10-08 22:26         ` Ananyev, Konstantin
2020-10-09 16:11           ` Burakov, Anatoly
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 04/10] power: add simple power management API and callback Liang Ma
2020-10-09 16:38         ` Ananyev, Konstantin
2020-10-09 16:47           ` Burakov, Anatoly
2020-10-09 16:51             ` Ananyev, Konstantin
2020-10-09 16:56               ` Burakov, Anatoly
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 05/10] net/ixgbe: implement power management API Liang Ma
2020-10-09 15:53         ` Ananyev, Konstantin
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 06/10] net/i40e: " Liang Ma
2020-10-09 16:01         ` Ananyev, Konstantin
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 07/10] net/ice: " Liang Ma
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 08/10] examples/l3fwd-power: enable PMD power mgmt Liang Ma
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 09/10] doc: update release notes for PMD power management Liang Ma
2020-10-02 14:11       ` [dpdk-dev] [PATCH v4 10/10] doc: update the programming guide " Liang Ma
2020-10-02 14:44       ` [dpdk-dev] [PATCH v4 01/10] eal: add new x86 cpuid support for WAITPKG Bruce Richardson
2020-10-08 22:08       ` Ananyev, Konstantin
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 " Anatoly Burakov
2020-10-09 17:06         ` Burakov, Anatoly
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 " Anatoly Burakov
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 " Anatoly Burakov
2020-10-23 17:17             ` [dpdk-dev] [PATCH v8 " Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 00/10] Add PMD power mgmt Liang Ma
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 0/9] " Liang Ma
2020-10-27 16:02                   ` Thomas Monjalon
2020-10-28 13:35                     ` Liang, Ma
2020-10-28 13:49                       ` Jerin Jacob
2020-10-28 14:21                         ` Thomas Monjalon
2020-10-28 14:57                           ` Ananyev, Konstantin
2020-10-28 15:14                             ` Jerin Jacob
2020-10-28 15:30                               ` Liang, Ma
2020-10-28 15:36                                 ` Jerin Jacob
2020-10-28 15:44                                   ` Liang, Ma
2020-10-28 16:01                                     ` Jerin Jacob
2020-10-28 16:21                                       ` Liang, Ma
2020-10-28 15:33                               ` Ananyev, Konstantin
2020-10-28 15:39                                 ` Jerin Jacob
2020-10-28 15:49                                   ` Ananyev, Konstantin
2020-10-28 15:57                                     ` Jerin Jacob
2020-10-28 16:38                                       ` Ananyev, Konstantin
2020-10-28 16:47                                       ` Liang, Ma
2020-10-28 16:54                                         ` McDaniel, Timothy
2020-10-29  9:19                                           ` Liang, Ma
2020-10-28 17:02                                         ` Ajit Khaparde
2020-10-28 18:10                                           ` Ananyev, Konstantin
2020-10-27 20:53                   ` Ajit Khaparde
2020-10-28 12:13                     ` Liang, Ma
2020-10-29 17:42                   ` Thomas Monjalon
2020-10-30  9:36                     ` Liang, Ma
2020-10-30  9:58                       ` Thomas Monjalon
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 1/9] eal: add new x86 cpuid support for WAITPKG Liang Ma
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 2/9] eal: add power management intrinsics Liang Ma
2020-10-29 17:39                   ` Thomas Monjalon
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure Liang Ma
2020-10-29 21:27                   ` David Marchand
2020-10-30 10:09                     ` Burakov, Anatoly
2020-10-30 10:14                       ` Thomas Monjalon
2020-10-30 13:37                         ` Burakov, Anatoly
2020-10-30 14:09                           ` Thomas Monjalon
2020-10-30 15:27                             ` Burakov, Anatoly
2020-10-30 15:44                               ` Thomas Monjalon
2020-10-30 16:36                                 ` Burakov, Anatoly
2020-10-30 16:50                                   ` Thomas Monjalon
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 4/9] ethdev: add simple power management API Liang Ma
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 5/9] power: add PMD power management API and callback Liang Ma
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 6/9] net/ixgbe: implement power management API Liang Ma
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 7/9] net/i40e: " Liang Ma
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 8/9] net/ice: " Liang Ma
2020-10-27 14:59                 ` [dpdk-dev] [PATCH v10 9/9] examples/l3fwd-power: enable PMD power mgmt Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 01/10] eal: add new x86 cpuid support for WAITPKG Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 02/10] eal: add power management intrinsics Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 03/10] eal: add intrinsics support check infrastructure Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API Liang Ma
2020-10-24 20:39                 ` Thomas Monjalon
2020-10-27 11:15                   ` Liang, Ma
2020-10-27 15:52                     ` Thomas Monjalon
2020-10-27 17:43                       ` Ananyev, Konstantin
2020-10-27 18:30                         ` Thomas Monjalon
2020-10-27 23:29                           ` Ananyev, Konstantin
2020-10-28  3:24                             ` Ajit Khaparde
2020-10-28 12:24                             ` Liang, Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 05/10] power: add PMD power management API and callback Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 06/10] net/ixgbe: implement power management API Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 07/10] net/i40e: " Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 08/10] net/ice: " Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 09/10] examples/l3fwd-power: enable PMD power mgmt Liang Ma
2020-10-23 23:06               ` [dpdk-dev] [PATCH v9 10/10] doc: update programmer's guide for power library Liang Ma
2020-10-24 20:49                 ` Thomas Monjalon
2020-10-27 11:04                   ` Liang, Ma
2020-10-23 17:20             ` [dpdk-dev] [PATCH v8 02/10] eal: add power management intrinsics Liang Ma
2020-10-23 17:21             ` [dpdk-dev] [PATCH v8 03/10] eal: add intrinsics support check infrastructure Liang Ma
2020-10-23 17:22             ` [dpdk-dev] [PATCH v8 04/10] ethdev: add simple power management API Liang Ma
2020-10-23 17:23             ` [dpdk-dev] [PATCH v8 05/10] power: add PMD power management API and callback Liang Ma
2020-10-23 17:26             ` [dpdk-dev] [PATCH v8 06/10] net/ixgbe: implement power management API Liang Ma
2020-10-23 17:26             ` [dpdk-dev] [PATCH v8 07/10] net/i40e: " Liang Ma
2020-10-23 17:27             ` [dpdk-dev] [PATCH v8 08/10] net/ice: " Liang Ma
2020-10-23 17:30             ` [dpdk-dev] [PATCH v8 09/10] examples/l3fwd-power: enable PMD power mgmt Liang Ma
2020-10-23 17:36             ` [dpdk-dev] [PATCH v8 10/10] doc: update programmer's guide for power library Liang Ma
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 02/10] eal: add power management intrinsics Anatoly Burakov
2020-10-15 12:06             ` Jerin Jacob
2020-10-15 13:16             ` Ferruh Yigit
2020-10-16  8:44               ` Ruifeng Wang
2020-10-15 16:43             ` Ananyev, Konstantin
2020-10-19 21:12             ` Thomas Monjalon
2020-10-20  2:49               ` Ruifeng Wang
2020-10-20  7:35                 ` Thomas Monjalon
2020-10-20 14:01                   ` David Hunt
2020-10-20 14:17                     ` David Hunt
2020-10-20 14:33                       ` Thomas Monjalon
2020-10-20 17:26                         ` Ananyev, Konstantin
2020-10-20 19:28                           ` Thomas Monjalon
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 03/10] eal: add intrinsics support check infrastructure Anatoly Burakov
2020-10-16  9:02             ` Ruifeng Wang
2020-10-16 11:21             ` Kinsella, Ray
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 04/10] ethdev: add simple power management API Anatoly Burakov
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 05/10] power: add PMD power management API and callback Anatoly Burakov
2020-10-15 16:52             ` Ananyev, Konstantin
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 06/10] net/ixgbe: implement power management API Anatoly Burakov
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 07/10] net/i40e: " Anatoly Burakov
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 08/10] net/ice: " Anatoly Burakov
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 09/10] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2020-10-15 12:04           ` [dpdk-dev] [PATCH v7 10/10] doc: update programmer's guide for power library Anatoly Burakov
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 02/10] eal: add power management intrinsics Anatoly Burakov
2020-10-14 17:48           ` Ananyev, Konstantin
2020-10-15 10:09             ` Burakov, Anatoly
2020-10-15 10:45               ` Burakov, Anatoly
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 03/10] eal: add intrinsics support check infrastructure Anatoly Burakov
2020-10-14 17:51           ` Ananyev, Konstantin
2020-10-14 17:59           ` Jerin Jacob
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 04/10] ethdev: add simple power management API Anatoly Burakov
2020-10-14 17:06           ` Ananyev, Konstantin
2020-10-15 11:29           ` Liang, Ma
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 05/10] power: add PMD power management API and callback Anatoly Burakov
2020-10-14 14:19           ` David Hunt
2020-10-14 18:41           ` Ananyev, Konstantin
2020-10-15 10:31             ` Burakov, Anatoly
2020-10-15 16:02               ` Ananyev, Konstantin
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 06/10] net/ixgbe: implement power management API Anatoly Burakov
2020-10-14 17:04           ` Wang, Haiyue
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 07/10] net/i40e: " Anatoly Burakov
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 08/10] net/ice: " Anatoly Burakov
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 09/10] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2020-10-14 14:24           ` David Hunt
2020-10-14 13:30         ` [dpdk-dev] [PATCH v6 10/10] doc: update programmer's guide for power library Anatoly Burakov
2020-10-14 14:27           ` David Hunt
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 02/10] eal: add power management intrinsics Anatoly Burakov
2020-10-09 16:09         ` Jerin Jacob
2020-10-09 16:24           ` Burakov, Anatoly
2020-10-12 19:47         ` David Christensen
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 03/10] eal: add intrinsics support check infrastructure Anatoly Burakov
2020-10-11 10:07         ` Jerin Jacob
2020-10-12  9:26           ` Burakov, Anatoly
2020-10-12 19:52         ` David Christensen
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 04/10] ethdev: add simple power management API Anatoly Burakov
2020-10-14  3:10         ` Guo, Jia
2020-10-14  9:07           ` Burakov, Anatoly
2020-10-14  9:15             ` Guo, Jia
2020-10-14  9:30               ` Burakov, Anatoly
2020-10-14  9:23             ` Bruce Richardson
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 05/10] power: add PMD power management API and callback Anatoly Burakov
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 06/10] net/ixgbe: implement power management API Anatoly Burakov
2020-10-12  7:46         ` Wang, Haiyue
2020-10-12  9:28           ` Burakov, Anatoly
2020-10-12  9:44           ` Burakov, Anatoly
2020-10-12 15:58             ` Wang, Haiyue
2020-10-12  8:09         ` Wang, Haiyue
2020-10-12  9:28           ` Burakov, Anatoly
2020-10-13  1:17             ` Wang, Haiyue
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 07/10] net/i40e: " Anatoly Burakov
2020-10-14  3:19         ` Guo, Jia
2020-10-14  9:08           ` Burakov, Anatoly
2020-10-14  9:17             ` Guo, Jia
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 08/10] net/ice: " Anatoly Burakov
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 09/10] examples/l3fwd-power: enable PMD power mgmt Anatoly Burakov
2020-10-09 16:02       ` [dpdk-dev] [PATCH v5 10/10] doc: update programmer's guide for power library Anatoly Burakov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d981e96f-b792-9db2-f0c0-ab0c0b997b00@intel.com \
    --to=anatoly.burakov@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=chris.macnamara@intel.com \
    --cc=david.hunt@intel.com \
    --cc=dev@dpdk.org \
    --cc=konstantin.ananyev@intel.com \
    --cc=liang.j.ma@intel.com \
    --cc=mdr@ashroe.eu \
    --cc=nhorman@tuxdriver.com \
    --cc=thomas@monjalon.net \
    --cc=timothy.mcdaniel@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).