* [dpdk-dev] Feature Discussion: CPU topology detection on FreeBSD @ 2021-01-28 4:07 Oscar Zhao 2021-01-29 10:05 ` Bruce Richardson 0 siblings, 1 reply; 6+ messages in thread From: Oscar Zhao @ 2021-01-28 4:07 UTC (permalink / raw) To: dev Hello dpdk devs, I would like to discuss the feasibility of implementing CPU topology detection on FreeBSD. Currently both eal_cpu_core_id() and eal_cpu_socket_id() in rte_eal library always return 0 on FreeBSD, making NUMA-aware development impossible without resorting to external libraries or system APIs. The CPU topology information is available via sysct kern.sched.topology_spec. The only issue is that the return value is an XML formatted object (see https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/). (https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/) I'm not sure how feasible it is to parse XML inside DPDK. On a side note, obtaining the physical NUMA node id of a core is easy - sysctl dev.cpu.[cpu#].%domain returns the corresponding node id but I have yet found a straightforward way to detect hyperthreads besides the XML thing. Oscar ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Feature Discussion: CPU topology detection on FreeBSD 2021-01-28 4:07 [dpdk-dev] Feature Discussion: CPU topology detection on FreeBSD Oscar Zhao @ 2021-01-29 10:05 ` Bruce Richardson 2021-01-29 12:52 ` Burakov, Anatoly 0 siblings, 1 reply; 6+ messages in thread From: Bruce Richardson @ 2021-01-29 10:05 UTC (permalink / raw) To: Oscar Zhao; +Cc: dev On Thu, Jan 28, 2021 at 04:07:07AM +0000, Oscar Zhao wrote: > Hello dpdk devs, > > I would like to discuss the feasibility of implementing CPU topology detection on FreeBSD. Currently both eal_cpu_core_id() and eal_cpu_socket_id() in rte_eal library always return 0 on FreeBSD, making NUMA-aware development impossible without resorting to external libraries or system APIs. > > The CPU topology information is available via sysct kern.sched.topology_spec. The only issue is that the return value is an XML formatted object (see https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/). (https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/) I'm not sure how feasible it is to parse XML inside DPDK. > > On a side note, obtaining the physical NUMA node id of a core is easy - sysctl dev.cpu.[cpu#].%domain returns the corresponding node id but I have yet found a straightforward way to detect hyperthreads besides the XML thing. > > Oscar Even adding in the numa node information would be a welcome start. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Feature Discussion: CPU topology detection on FreeBSD 2021-01-29 10:05 ` Bruce Richardson @ 2021-01-29 12:52 ` Burakov, Anatoly 2021-01-29 13:45 ` Bruce Richardson 2021-01-29 22:24 ` Oscar Zhao 0 siblings, 2 replies; 6+ messages in thread From: Burakov, Anatoly @ 2021-01-29 12:52 UTC (permalink / raw) To: Bruce Richardson, Oscar Zhao; +Cc: dev On 29-Jan-21 10:05 AM, Bruce Richardson wrote: > On Thu, Jan 28, 2021 at 04:07:07AM +0000, Oscar Zhao wrote: >> Hello dpdk devs, >> >> I would like to discuss the feasibility of implementing CPU topology detection on FreeBSD. Currently both eal_cpu_core_id() and eal_cpu_socket_id() in rte_eal library always return 0 on FreeBSD, making NUMA-aware development impossible without resorting to external libraries or system APIs. >> >> The CPU topology information is available via sysct kern.sched.topology_spec. The only issue is that the return value is an XML formatted object (see https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/). (https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/) I'm not sure how feasible it is to parse XML inside DPDK. >> >> On a side note, obtaining the physical NUMA node id of a core is easy - sysctl dev.cpu.[cpu#].%domain returns the corresponding node id but I have yet found a straightforward way to detect hyperthreads besides the XML thing. >> >> Oscar > > Even adding in the numa node information would be a welcome start. > There's little value in it though, because as far as i'm aware we can't allocate memory on specific NUMA nodes in FreeBSD. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Feature Discussion: CPU topology detection on FreeBSD 2021-01-29 12:52 ` Burakov, Anatoly @ 2021-01-29 13:45 ` Bruce Richardson 2021-02-10 16:01 ` Burakov, Anatoly 2021-01-29 22:24 ` Oscar Zhao 1 sibling, 1 reply; 6+ messages in thread From: Bruce Richardson @ 2021-01-29 13:45 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: Oscar Zhao, dev On Fri, Jan 29, 2021 at 12:52:50PM +0000, Burakov, Anatoly wrote: > On 29-Jan-21 10:05 AM, Bruce Richardson wrote: > > On Thu, Jan 28, 2021 at 04:07:07AM +0000, Oscar Zhao wrote: > > > Hello dpdk devs, > > > > > > I would like to discuss the feasibility of implementing CPU topology detection on FreeBSD. Currently both eal_cpu_core_id() and eal_cpu_socket_id() in rte_eal library always return 0 on FreeBSD, making NUMA-aware development impossible without resorting to external libraries or system APIs. > > > > > > The CPU topology information is available via sysct kern.sched.topology_spec. The only issue is that the return value is an XML formatted object (see https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/). (https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/) I'm not sure how feasible it is to parse XML inside DPDK. > > > > > > On a side note, obtaining the physical NUMA node id of a core is easy - sysctl dev.cpu.[cpu#].%domain returns the corresponding node id but I have yet found a straightforward way to detect hyperthreads besides the XML thing. > > > > > > Oscar > > > > Even adding in the numa node information would be a welcome start. > > > > There's little value in it though, because as far as i'm aware we can't > allocate memory on specific NUMA nodes in FreeBSD. > That is because the NUMA support in FreeBSD was not in place when we first did the DPDK port. However, I still see no reason not to start adding it now, even if it is only for information purposes at this point. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Feature Discussion: CPU topology detection on FreeBSD 2021-01-29 13:45 ` Bruce Richardson @ 2021-02-10 16:01 ` Burakov, Anatoly 0 siblings, 0 replies; 6+ messages in thread From: Burakov, Anatoly @ 2021-02-10 16:01 UTC (permalink / raw) To: Bruce Richardson; +Cc: Oscar Zhao, dev On 29-Jan-21 1:45 PM, Bruce Richardson wrote: > On Fri, Jan 29, 2021 at 12:52:50PM +0000, Burakov, Anatoly wrote: >> On 29-Jan-21 10:05 AM, Bruce Richardson wrote: >>> On Thu, Jan 28, 2021 at 04:07:07AM +0000, Oscar Zhao wrote: >>>> Hello dpdk devs, >>>> >>>> I would like to discuss the feasibility of implementing CPU topology detection on FreeBSD. Currently both eal_cpu_core_id() and eal_cpu_socket_id() in rte_eal library always return 0 on FreeBSD, making NUMA-aware development impossible without resorting to external libraries or system APIs. >>>> >>>> The CPU topology information is available via sysct kern.sched.topology_spec. The only issue is that the return value is an XML formatted object (see https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/). (https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299/) I'm not sure how feasible it is to parse XML inside DPDK. >>>> >>>> On a side note, obtaining the physical NUMA node id of a core is easy - sysctl dev.cpu.[cpu#].%domain returns the corresponding node id but I have yet found a straightforward way to detect hyperthreads besides the XML thing. >>>> >>>> Oscar >>> >>> Even adding in the numa node information would be a welcome start. >>> >> >> There's little value in it though, because as far as i'm aware we can't >> allocate memory on specific NUMA nodes in FreeBSD. >> > > That is because the NUMA support in FreeBSD was not in place when we first > did the DPDK port. However, I still see no reason not to start adding it > now, even if it is only for information purposes at this point. > A lot of DPDK apps will fail to work in this case, because most of them will allocate with an explicit NUMA node. So, if you have a NUMA node assigned to an lcore but can only allocate from NUMA node 0, any allocation explicitly targeting NUMA node 1 will not succeed. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Feature Discussion: CPU topology detection on FreeBSD 2021-01-29 12:52 ` Burakov, Anatoly 2021-01-29 13:45 ` Bruce Richardson @ 2021-01-29 22:24 ` Oscar Zhao 1 sibling, 0 replies; 6+ messages in thread From: Oscar Zhao @ 2021-01-29 22:24 UTC (permalink / raw) To: Bruce Richardson, Burakov, Anatoly; +Cc: dev FreeBSD rolled their own NUMA memory APIs on 12.0 (see https://www.freebsd.org/cgi/man.cgi?query=cpuset_getdomain&sektion=2&apropos=0&manpath=FreeBSD+12.2-RELEASE+and+Ports). These syscalls allow binding of a process/thread/jail memory allocation to a specific NUMA domain. I think the first step is to support correct NUMA detection. The "de facto" way of doing that is parsing the XML output from the cpu topology sysctl. I'm not sure how viable it is in DPDK without introducing external dependencies. I've also been playing with the hwloc library which supposedly offers portable topology detection. If both options are not feasible maybe something can be done on the FreeBSD side first. January 29, 2021 8:45 AM, "Bruce Richardson" <bruce.richardson@intel.com> wrote: > On Fri, Jan 29, 2021 at 12:52:50PM +0000, Burakov, Anatoly wrote: > >> On 29-Jan-21 10:05 AM, Bruce Richardson wrote: >> On Thu, Jan 28, 2021 at 04:07:07AM +0000, Oscar Zhao wrote: >>> Hello dpdk devs, >>> >>> I would like to discuss the feasibility of implementing CPU topology detection on FreeBSD. >> Currently both eal_cpu_core_id() and eal_cpu_socket_id() in rte_eal library always return 0 on >> FreeBSD, making NUMA-aware development impossible without resorting to external libraries or system >> APIs. >>> >>> The CPU topology information is available via sysct kern.sched.topology_spec. The only issue is >> that the return value is an XML formatted object (see >> https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299). >> (https://forums.freebsd.org/threads/number-of-cpus-and-cores.41299) I'm not sure how feasible it is >> to parse XML inside DPDK. >>> >>> On a side note, obtaining the physical NUMA node id of a core is easy - sysctl >> dev.cpu.[cpu#].%domain returns the corresponding node id but I have yet found a straightforward way >> to detect hyperthreads besides the XML thing. >>> >>> Oscar >> >> Even adding in the numa node information would be a welcome start. >> >> There's little value in it though, because as far as i'm aware we can't >> allocate memory on specific NUMA nodes in FreeBSD. > > That is because the NUMA support in FreeBSD was not in place when we first > did the DPDK port. However, I still see no reason not to start adding it > now, even if it is only for information purposes at this point. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-02-10 16:02 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-28 4:07 [dpdk-dev] Feature Discussion: CPU topology detection on FreeBSD Oscar Zhao 2021-01-29 10:05 ` Bruce Richardson 2021-01-29 12:52 ` Burakov, Anatoly 2021-01-29 13:45 ` Bruce Richardson 2021-02-10 16:01 ` Burakov, Anatoly 2021-01-29 22:24 ` Oscar Zhao
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).