From: "Morten Brørup" <mb@smartsharesystems.com>
To: "Bruce Richardson" <bruce.richardson@intel.com>
Cc: "Didier Pallard" <didier.pallard@6wind.com>,
"Akhil Goyal" <gakhil@marvell.com>,
"Fan Zhang" <fanzhang.oss@gmail.com>,
"Olivier Matz" <olivier.matz@6wind.com>, <thomas@monjalon.net>,
<dev@dpdk.org>
Subject: RE: [RFC] Fix cryptodev socket id for devices on unknown NUMA node
Date: Wed, 18 Jan 2023 09:52:14 +0100 [thread overview]
Message-ID: <98CBD80474FA8B44BF855DF32C47DC35D87684@smartserver.smartshare.dk> (raw)
In-Reply-To: <Y8aptk4Lvsn+QUGg@bricha3-MOBL.ger.corp.intel.com>
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Tuesday, 17 January 2023 14.59
>
> On Tue, Jan 17, 2023 at 02:36:21PM +0100, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Tuesday, 17 January 2023 14.04
> > >
> > > On Tue, Jan 17, 2023 at 12:32:14PM +0100, Morten Brørup wrote:
> > > > > From: Didier Pallard [mailto:didier.pallard@6wind.com]
> > > > > Sent: Tuesday, 17 January 2023 11.17
> > > > >
> > > > > Since DPDK 22.11 and below commit:
> > > > >
> > >
> https://git.dpdk.org/dpdk/commit/?id=7dcd73e37965ba0bfa430efeac362fe183
> > > > > ed0ae2
> > > > > rte_cryptodev_socket_id() could return an incorrect value of
> 255.
> > > > > Problem has been seen during configuration of the qat device
> > > > > on an Atom C3000 architecture. On this arch, PCI is not
> depending
> > > on
> > > > > any numa socket, causing device numa_node to be equal to
> > > SOCKET_ID_ANY.
> > > >
> > > > Disclaimer: I'm not up to speed with this topic or patch, so feel
> > > free to ignore my comments here! I'm only speaking up because I
> fear we
> > > are increasing the risk of bugs here. But again, please bear with
> me,
> > > if I have totally misunderstood this!
> > > >
> > > > I was under the impression that single-socket systems used
> socket_id
> > > 0 as default. How can the PCI bus (or QAT device) not depend on any
> > > socket? It must be connected somewhere.
> > > >
> > > > Doesn't assigning socket_id = -1 for devices (QAT or anything
> else)
> > > introduce a big risk of bugs, e.g. in comparisons? The special
> > > socket_id value -1 should have only two meanings: 1) return value
> > > "error", or 2) input value "any". Now it also can mean 3)
> "unknown"?
> > > How do comparison functions work for that... is "any" == "unknown"?
> And
> > > does searching for "0" match "unknown"? It might, or might not, but
> > > searching for "any" does match "0". And how about searching for
> > > "unknown", if such a value is propagate around in the system.
> > > >
> > > > And if we started considering socket_id == -1 valid with that
> patch,
> > > should the return type of rte_socket_id(void) be signed instead of
> > > unsigned?
> > > >
> > > The issue here is that not all PCI endpoints connect directly to a
> > > socket,
> > > some connect to the chipset instead, and so do not have any numa
> > > affinity.
> > > That was the original meaning of the "-1" value, and it came about
> from
> > > an
> > > era before we had on-die PCI endpoints.
> >
> > Thank you for elaborating, Bruce. Now I get it!
> >
> > Then it does make sense instantiating devices with socket_id = -1.
> >
> > A minor detail: SOCKET_ID_ANY is defined in
> lib/eal/include/rte_memory.h [1]. Since it is being used for other
> purposes than memory allocation, it could move to a more central
> location. No good ideas from me, but perhaps memory and devices have an
> appropriate header file in common.
> >
> > [1]:
> https://elixir.bootlin.com/dpdk/latest/source/lib/eal/include/rte_memor
> y.h#L38
> >
> > And the multiple purposes of SOCKET_ID_ANY (e.g. what you just
> described) could be mentioned as a comment with its definition.
> >
>
> Agreed.
>
> > And I'm still worried about the risk of comparison bugs, e.g. when
> requesting allocation of a device resource, you cannot specify a
> preference for a device that is connected to the chipset, because the
> SOCKET_ID_ANY in the allocation request would be interpreted as "any
> socket" instead of "no socket".
> >
> > Although that might just be me worrying too much. ;-)
> >
> No, I think you may be on to something. I wonder if it would break a
> lot of
> things to define another magic constant for SOCKET_NONE (or
> SOCKET_ID_NONE)
> to cover the case where a device is not connected to a socket. That
> woudl
> allow SOCKET_ID_ANY to have a single meaning.
I hate polluting an API with workarounds like reusing SOCKET_ID_ANY for alternative purposes, so a roadmap for introducing SOCKET_ID_NONE as a pseudo numa node should be defined before the current workaround causes too much damage.
E.g. here's a discussion [1] referring to an application using rte_eth_dev_socket_id() to improve performance.
[1]: http://inbox.dpdk.org/dev/SJ0PR11MB4783BE2C718D03E3C447B7DD80C79@SJ0PR11MB4783.namprd11.prod.outlook.com/
PS: High level changes like this (support for devices connected to the chipset instead of a specific cpu socket) should go on the roadmap page.
prev parent reply other threads:[~2023-01-18 8:52 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-17 10:16 Didier Pallard
2023-01-17 11:32 ` Morten Brørup
2023-01-17 13:03 ` Bruce Richardson
2023-01-17 13:36 ` Morten Brørup
2023-01-17 13:59 ` Bruce Richardson
2023-01-18 8:52 ` Morten Brørup [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=98CBD80474FA8B44BF855DF32C47DC35D87684@smartserver.smartshare.dk \
--to=mb@smartsharesystems.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=didier.pallard@6wind.com \
--cc=fanzhang.oss@gmail.com \
--cc=gakhil@marvell.com \
--cc=olivier.matz@6wind.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).