From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9DE3B4406B; Sun, 19 May 2024 17:14:00 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1F2CD402A3; Sun, 19 May 2024 17:14:00 +0200 (CEST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mails.dpdk.org (Postfix) with ESMTP id 3757C402A3 for ; Sun, 19 May 2024 17:13:58 +0200 (CEST) Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1ed41eb3382so53139735ad.0 for ; Sun, 19 May 2024 08:13:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1716131637; x=1716736437; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=ok8q4102rNKd6CryoheQCrCsmmzDxj35+jZo2PSxNpE=; b=OpoVf49D96i8oFUeImMfIpjEk4sSQLnMABCDDJ/paZbq9rI3EC+1W327ITPrkQmZ3y H34OFs8GswbltLEbyYN1PYST8jmv9se9FvKXMNgsoEdImjo0dYgL+MFE9ytr/u1sWA2E 93nsOxkVF+MSK+gmvdvZgCCOJ6n8LgNwJUtCVpNT+h49CwceAhT3hz/nz82O2iMOKXN3 eKtmRzJNRk7oOdQVJPmWWKRy9x32KRBJB+umhg8YsVc8BAygGfjWeDOzMhZUcIJ/Wt7Q Xh4rCq8WgfHVwZq/bBGIgYBAg5C/LASFaHUAXAFil8MnUSWTxXNgj24wwqAeKaIAjbst 7REw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716131637; x=1716736437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ok8q4102rNKd6CryoheQCrCsmmzDxj35+jZo2PSxNpE=; b=m1r8ndp3c3Y8dUh6cN9sgSKR2Yva1jJr/EoGzszQVJ2+iuoRVICn1fRysAypO/zc5I F4SkGAT5StGbMeko5VPLKAiBPvBqDH1qKTnrvAkurjuNmolrOWvQmFsbKRXppMKQWjA+ 3jsu1Fzcrw0pfP7mTGnQsqoI65OtUbrwKrg2RP+/3l3yRSiRy40dVLD+8vLZKHkb+FF0 3Vg4kKsgjQxTOW+POc+zT2KxhXCEG8NsDSgV0WlBpTSq7SQx6STbymjyzYM7SCcN9nj9 RPmE0W+x/+M0IuCuK+KqXonEfNNqXfCkD3ThG6fNJkLT5ptZUrV+RnG2+W5UrIpuu6NQ oyQg== X-Forwarded-Encrypted: i=1; AJvYcCWoqcXn/dw3VFlRwj59HfwfOlHqQIL2g1WEEhpOWSFrHs+Xsde4w1e9J/iv1QknHGhHqQnfKMnE1ozS+Ak= X-Gm-Message-State: AOJu0Yyco0Gq5SPvBlLSOlsMgjvgn4VDrLylwUahsEl/pRIcWwNJNCP/ EHBv97++v2WZoZ05M1D9EWDx2z0YzkxjdmEM/WQQMFYXT1d/SAjpLtcjspxV4TYXnyyysubElBL jw+A= X-Google-Smtp-Source: AGHT+IE5AZsx+BwvdM3eElqXUGwAaDa/Ovay7T1HChJxQUcGk+uN5swTx+n+vgpK/DA1ViogIgKbEA== X-Received: by 2002:a17:90b:4ac7:b0:2bd:6d5b:a100 with SMTP id 98e67ed59e1d1-2bd6d5ba539mr2265934a91.49.1716131637007; Sun, 19 May 2024 08:13:57 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b96f314fffsm10466213a91.48.2024.05.19.08.13.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 May 2024 08:13:56 -0700 (PDT) Date: Sun, 19 May 2024 08:13:55 -0700 From: Stephen Hemminger To: Morten =?UTF-8?B?QnLDuHJ1cA==?= Cc: "Honnappa Nagarahalli" , , "nd" , "Richardson, Bruce" Subject: Re: [PATCH v6 1/9] eal: generic 64 bit counter Message-ID: <20240519081355.6b51c3ba@hermes.local> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9F47C@smartserver.smartshare.dk> References: <20240510050507.14381-1-stephen@networkplumber.org> <20240517001302.65514-1-stephen@networkplumber.org> <20240517001302.65514-2-stephen@networkplumber.org> <48ED00A3-1CCA-4F64-ACC3-CA1F0D2B9378@arm.com> <20240516203037.73ec13e1@hermes.local> <248621D2-C402-4DDE-92D8-F5377E816533@arm.com> <98CBD80474FA8B44BF855DF32C47DC35E9F476@smartserver.smartshare.dk> <20240517091801.17fdef27@hermes.local> <98CBD80474FA8B44BF855DF32C47DC35E9F47C@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Sat, 18 May 2024 16:00:55 +0200 Morten Br=C3=B8rup wrote: > > From: Stephen Hemminger [mailto:stephen@networkplumber.org] > > Sent: Friday, 17 May 2024 18.18 > >=20 > > On Fri, 17 May 2024 08:44:42 +0200 > > Morten Br=C3=B8rup wrote: > > =20 > > > > From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com] > > > > Sent: Friday, 17 May 2024 06.27 > > > > > > > > + Bruce for feedback on x86 architecture > > > > =20 > > > > > On May 16, 2024, at 10:30=E2=80=AFPM, Stephen Hemminger =20 > > =20 > > > > wrote: =20 > > > > > > > > > > On Fri, 17 May 2024 02:45:12 +0000 > > > > > Honnappa Nagarahalli wrote: > > > > > =20 > > > > >>> + * A counter is 64 bit value that is safe from split read/write > > > > >>> + * on 32 bit platforms. It assumes that only one cpu at a time= =20 > > > > >> If we are defining the counter in this manner, then =20 > > implementation cannot =20 > > > > be generic. I think architectures will have constraints if they hav= e =20 > > to ensure =20 > > > > the 64b variables are not split. =20 > > > > >> > > > > >> I think we at least need the counter to be aligned on 8B boundar= y =20 > > to have =20 > > > > generic code. =20 > > > > > > > > > > The C standard has always guaranteed that read and write to =20 > > unsigned log =20 > > > > will not be split. > > > > As I understand, this is true only if the variable is an atomic =20 > > variable. If =20 > > > > not, there is no difference between atomic variables and non-atomic= =20 > > variables. =20 > > > > =20 > > > > > Therefore if arch is 64 bit native there is no need for atomics = =20 > > > > At least on ARM architecture, if the variable is not aligned on 8B = =20 > > boundary, =20 > > > > the load or store are not atomic. I am sure it is the same on other > > > > architectures. =20 > >=20 > > After reading this: Who's afraid of a big bad optimizing compiler? > > https://lwn.net/Articles/793253/ =20 >=20 > Very interesting article! >=20 > >=20 > > Looks like you are right, and atomic or read/write once is required. =20 >=20 > I don't like the performance tradeoff (for 64 bit architectures) in the v= 7 patch. > For single-tread updated counters, we MUST have a high performance counte= r_add(), not using atomic read-modify-write. The fundamental issue is that for SW drivers, having a totally safe reset f= unction requires an atomic operation in the fast path for increment. Otherwise the following= (highly unlikely) race is possible: =09 CPU A CPU B load counter (value =3D X) store counter =3D 0 store counter (value =3D X + 1) >=20 > IMO calling counter_fetch() MUST be possible from another thread. > This requires that the fast path thread stores the counter atomically (or= using volatile), to ensure that the counter is not only kept in a CPU regi= ster, but stored in memory visible by other threads. >=20 > For counter_reset(), we have multiple options: > 0. Don't support counter resetting. Not really on option? We could reject it in the SW drivers. But better not to. > 1. Only allow calling counter_reset() in the fast path thread updating th= e counter. This introduces no further requirements. Not a good restriction > 2. Allow calling counter_reset() from another thread, thereby zeroing the= "counter" variable. This introduces a requirement for the "counter" to be = thread-safe, so counter_add() must atomically read-modify-write the counter= , which has a performance cost. > 3. Allow calling counter_reset() from another thread, and introduce an "o= ffset" variable for counter_fetch() and counter_reset() to provide thread-s= afe fetch/reset from other threads, using the consume-release pattern. Too confusing >=20 > I don't like option 2. > I consider counters too important and frequently used in the fast path, t= o compromise on performance for counters. Agree. >=20 > For counters updated by multiple fast path threads, atomic_fetch_add_expl= icit() of the "counter" variable seems unavoidable. Not at all worried about overhead in slow (fetch) path. >=20 > > Perhaps introducing rte_read_once and rte_write_once is good idea? > > Several drivers are implementing it already. =20 >=20 > The read_once/write_once are easier to use, but they lack the flexibility= (regarding barriers and locking) provided by their atomic_explicit alterna= tives, which will impact performance in some use cases. They solve the compiler reordering problem but do nothing about cpu orderin= g. Also, there is no such increment. >=20 > We should strive for the highest possible performance, which means that w= e shouldn't introduce functions or design patterns preventing this. > Please note: Security vs. performance is another matter - we certainly do= n't want to promote insecure code for the benefit of performance. But for "= ease of coding" vs. performance, I prefer performance. >=20 > That said, I agree that introducing rte_read/write_once functions for use= by drivers to access hardware makes sense, to eliminate copy-paste variant= s in drivers. > But how can we prevent such functions from being used for other purposes,= where atomic_explicit should be used? >=20 Looking at x86 result on godbolt shows that using atomic only adds a single= locked operation. Perhaps this can be ameliorated by doing bulk add at end of loop, like many= drivers were already.