From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
To: "Mattias Rönnblom" <hofors@lysator.liu.se>
Cc: "Ferruh Yigit" <ferruh.yigit@amd.com>,
"Mattias Rönnblom" <mattias.ronnblom@ericsson.com>,
"John W . Linville" <linville@tuxdriver.com>,
"dev@dpdk.org" <dev@dpdk.org>,
"Tyler Retzlaff" <roretzla@linux.microsoft.com>, nd <nd@arm.com>,
"Honnappa Nagarahalli" <Honnappa.Nagarahalli@arm.com>
Subject: Re: [PATCH] net/af_packet: cache align Rx/Tx structs
Date: Wed, 24 Apr 2024 00:27:37 +0000 [thread overview]
Message-ID: <4E55A056-C269-4DEA-B702-1979BF66E574@arm.com> (raw)
In-Reply-To: <63dbb564-61f6-4d9f-9c13-4a21f5e97dc9@lysator.liu.se>
> On Apr 23, 2024, at 3:56 PM, Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>
> On 2024-04-23 13:15, Ferruh Yigit wrote:
>> On 4/23/2024 10:08 AM, Mattias Rönnblom wrote:
>>> Cache align Rx and Tx queue struct to avoid false sharing.
>>>
>>> RX struct happens to be 64 bytes on x86_64 already, so cache alignment
>>> makes no change there, but it does on 32-bit ISAs.
>>>
>>> TX struct is 56 bytes on x86_64.
>>>
>> Hi Mattias,
>> No objection to the patch. Is the improvement theoretical or do you
>> measure any improvement practically, if so how much is the improvement?
>
> I didn't run any benchmarks.
>
> Two cores storing to a (falsely) shared cache line on a per-packet basis is going to be very expensive, at least for "light touch" applications.
>
>>> Both structs keep counters, and in the RX case they are updated even
>>> for empty polls.
>>>
>> Do you think does it help if move 'rx_pkts' & 'rx_bytes' update within
>> the loop?
>
> No, why? Wouldn't that be worse? Especially since rx_pkts and rx_bytes are declared volatile, so you are forcing a load-modify-store cycle for every increment.
>
> I would drop "volatile", or replace it with an atomic (although *not* use an atomic add for incrementing, but rather atomic load + <n> non-atomic adds + atomic store).
(Slightly unrelated discussion)
Does the atomic load + increment + atomic store help in a non-contended case like this? Some platforms have optimizations for atomic-increments as well which would be missed.
>
>>> Signed-off-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
>>> ---
>>> drivers/net/af_packet/rte_eth_af_packet.c | 5 +++--
>>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
>>> index 397a32db58..28aeb7d08e 100644
>>> --- a/drivers/net/af_packet/rte_eth_af_packet.c
>>> +++ b/drivers/net/af_packet/rte_eth_af_packet.c
>>> @@ -6,6 +6,7 @@
>>> * All rights reserved.
>>> */
>>> +#include <rte_common.h>
>>> #include <rte_string_fns.h>
>>> #include <rte_mbuf.h>
>>> #include <ethdev_driver.h>
>>> @@ -53,7 +54,7 @@ struct pkt_rx_queue {
>>> volatile unsigned long rx_pkts;
>>> volatile unsigned long rx_bytes;
>>> -};
>>> +} __rte_cache_aligned;
>>>
>> Latest location for '__rte_cache_aligned' tag is at the beginning of the
>> struct [1], so something like:
>> `struct __rte_cache_aligned pkt_rx_queue {`
>> [1]
>> https://patchwork.dpdk.org/project/dpdk/list/?series=31746&state=%2A&archive=both
next prev parent reply other threads:[~2024-04-24 0:27 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-23 9:08 Mattias Rönnblom
2024-04-23 11:15 ` Ferruh Yigit
2024-04-23 20:56 ` Mattias Rönnblom
2024-04-24 0:27 ` Honnappa Nagarahalli [this message]
2024-04-24 6:28 ` Mattias Rönnblom
2024-04-24 10:21 ` Ferruh Yigit
2024-04-24 10:28 ` Bruce Richardson
2024-04-24 18:02 ` Ferruh Yigit
2024-04-24 11:57 ` Mattias Rönnblom
2024-04-24 17:50 ` Ferruh Yigit
2024-04-24 19:13 ` Stephen Hemminger
2024-04-24 22:27 ` Mattias Rönnblom
2024-04-24 23:55 ` Stephen Hemminger
2024-04-25 9:26 ` Mattias Rönnblom
2024-04-25 9:49 ` Morten Brørup
2024-04-25 14:04 ` Ferruh Yigit
2024-04-25 15:06 ` Mattias Rönnblom
2024-04-25 16:21 ` Ferruh Yigit
2024-04-25 15:07 ` Stephen Hemminger
2024-04-25 14:08 ` Ferruh Yigit
2024-04-25 15:08 ` Mattias Rönnblom
2024-04-25 15:35 ` Ferruh Yigit
2024-04-26 7:25 ` Mattias Rönnblom
2024-04-26 7:38 ` Mattias Rönnblom
2024-04-26 8:27 ` Ferruh Yigit
2024-04-26 10:20 ` Mattias Rönnblom
2024-04-26 9:05 ` [PATCH v3] " Mattias Rönnblom
2024-04-26 9:22 ` Morten Brørup
2024-04-26 15:10 ` Stephen Hemminger
2024-04-26 15:41 ` Tyler Retzlaff
2024-04-29 8:46 ` Ferruh Yigit
2024-04-26 21:27 ` [PATCH] " Patrick Robb
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E55A056-C269-4DEA-B702-1979BF66E574@arm.com \
--to=honnappa.nagarahalli@arm.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@amd.com \
--cc=hofors@lysator.liu.se \
--cc=linville@tuxdriver.com \
--cc=mattias.ronnblom@ericsson.com \
--cc=nd@arm.com \
--cc=roretzla@linux.microsoft.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).