quick thread in DLB2

quick thread in DLB2

[Thomas Monjalon]

Hello Abdullah,

In the DLB2 code, I see a thread is created for a single operation:
In drivers/event/dlb2/pf/base/dlb2_resource.c
pthread_create(&pthread, NULL, &dlb2_pp_profile_func, &dlb2_thread_data[i]);
and just after:
pthread_join(pthread, NULL);

Can we avoid creating this thread?
I guess no, because it must spawn on a specific CPU.

Re: quick thread in DLB2

[Sevincer, Abdullah]

[-- Attachment #1: Type: text/plain, Size: 715 bytes --]

Hi Thomas,

That's right we need to create threads on specific CPUs.
________________________________
From: Thomas Monjalon <thomas@monjalon.net>
Sent: Friday, September 1, 2023 7:09 AM
To: Sevincer, Abdullah <abdullah.sevincer@intel.com>
Cc: dev@dpdk.org <dev@dpdk.org>; Tyler Retzlaff <roretzla@linux.microsoft.com>
Subject: quick thread in DLB2

Hello Abdullah,

In the DLB2 code, I see a thread is created for a single operation:
In drivers/event/dlb2/pf/base/dlb2_resource.c
pthread_create(&pthread, NULL, &dlb2_pp_profile_func, &dlb2_thread_data[i]);
and just after:
pthread_join(pthread, NULL);

Can we avoid creating this thread?
I guess no, because it must spawn on a specific CPU.



[-- Attachment #2: Type: text/html, Size: 1575 bytes --]

Re: quick thread in DLB2

[Stephen Hemminger]

On Fri, 01 Sep 2023 16:08:48 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:

> Hello Abdullah,
> 
> In the DLB2 code, I see a thread is created for a single operation:
> In drivers/event/dlb2/pf/base/dlb2_resource.c
> pthread_create(&pthread, NULL, &dlb2_pp_profile_func, &dlb2_thread_data[i]);
> and just after:
> pthread_join(pthread, NULL);
> 
> Can we avoid creating this thread?
> I guess no, because it must spawn on a specific CPU.
> 
> 

The per thread data seems to break lots of expectations in EAL.
It all seems to be about capturing the number of cycles on different cores.
Looks like a mess.

RE: quick thread in DLB2

[Sevincer, Abdullah]

Hi Stephen,
It is probing ports for best CPU. Yes it collects cycles. We may rework in the future.
Open to suggestions.

-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Wednesday, September 6, 2023 12:45 PM
To: Thomas Monjalon <thomas@monjalon.net>
Cc: Sevincer, Abdullah <abdullah.sevincer@intel.com>; dev@dpdk.org; Tyler Retzlaff <roretzla@linux.microsoft.com>
Subject: Re: quick thread in DLB2

On Fri, 01 Sep 2023 16:08:48 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:

> Hello Abdullah,
> 
> In the DLB2 code, I see a thread is created for a single operation:
> In drivers/event/dlb2/pf/base/dlb2_resource.c
> pthread_create(&pthread, NULL, &dlb2_pp_profile_func, 
> &dlb2_thread_data[i]); and just after:
> pthread_join(pthread, NULL);
> 
> Can we avoid creating this thread?
> I guess no, because it must spawn on a specific CPU.
> 
> 

The per thread data seems to break lots of expectations in EAL.
It all seems to be about capturing the number of cycles on different cores.
Looks like a mess.

Re: quick thread in DLB2

[Stephen Hemminger]

On Thu, 7 Sep 2023 22:09:06 +0000
"Sevincer, Abdullah" <abdullah.sevincer@intel.com> wrote:

> Hi Stephen,
> It is probing ports for best CPU. Yes it collects cycles. We may rework in the future.
> Open to suggestions.

Why is this device a special snowflake? All threading should be using the existing
infrastructure of service and control threads. Doing it this way looks like a special
case optimization for testpmd.  What if the user is using a multi-stage pipeline,
flow rules, secondary processes, rings etc. Look at some of the real applications
using DPDK.  And talk to some of the network equipment vendors.

Re: quick thread in DLB2

[Mattias Rönnblom]

On 2023-09-08 00:09, Sevincer, Abdullah wrote:
> Hi Stephen,
> It is probing ports for best CPU. Yes it collects cycles. We may rework in the future.

Best, in what sense? Is this some kind of topology exploration? One DLB 
port being closer to (cheaper to access for) certain cores?

> Open to suggestions.
> 
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, September 6, 2023 12:45 PM
> To: Thomas Monjalon <thomas@monjalon.net>
> Cc: Sevincer, Abdullah <abdullah.sevincer@intel.com>; dev@dpdk.org; Tyler Retzlaff <roretzla@linux.microsoft.com>
> Subject: Re: quick thread in DLB2
> 
> On Fri, 01 Sep 2023 16:08:48 +0200
> Thomas Monjalon <thomas@monjalon.net> wrote:
> 
>> Hello Abdullah,
>>
>> In the DLB2 code, I see a thread is created for a single operation:
>> In drivers/event/dlb2/pf/base/dlb2_resource.c
>> pthread_create(&pthread, NULL, &dlb2_pp_profile_func,
>> &dlb2_thread_data[i]); and just after:
>> pthread_join(pthread, NULL);
>>
>> Can we avoid creating this thread?
>> I guess no, because it must spawn on a specific CPU.
>>
>>
> 
> The per thread data seems to break lots of expectations in EAL.
> It all seems to be about capturing the number of cycles on different cores.
> Looks like a mess.

RE: quick thread in DLB2

[Sevincer, Abdullah]

Mattias,
Yes that’s correct.


-----Original Message-----
From: Mattias Rönnblom <hofors@lysator.liu.se> 
Sent: Friday, September 8, 2023 12:28 AM
To: Sevincer, Abdullah <abdullah.sevincer@intel.com>; Stephen Hemminger <stephen@networkplumber.org>; Thomas Monjalon <thomas@monjalon.net>
Cc: dev@dpdk.org; Tyler Retzlaff <roretzla@linux.microsoft.com>
Subject: Re: quick thread in DLB2

On 2023-09-08 00:09, Sevincer, Abdullah wrote:
> Hi Stephen,
> It is probing ports for best CPU. Yes it collects cycles. We may rework in the future.

Best, in what sense? Is this some kind of topology exploration? One DLB port being closer to (cheaper to access for) certain cores?

> Open to suggestions.
> 
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, September 6, 2023 12:45 PM
> To: Thomas Monjalon <thomas@monjalon.net>
> Cc: Sevincer, Abdullah <abdullah.sevincer@intel.com>; dev@dpdk.org; 
> Tyler Retzlaff <roretzla@linux.microsoft.com>
> Subject: Re: quick thread in DLB2
> 
> On Fri, 01 Sep 2023 16:08:48 +0200
> Thomas Monjalon <thomas@monjalon.net> wrote:
> 
>> Hello Abdullah,
>>
>> In the DLB2 code, I see a thread is created for a single operation:
>> In drivers/event/dlb2/pf/base/dlb2_resource.c
>> pthread_create(&pthread, NULL, &dlb2_pp_profile_func, 
>> &dlb2_thread_data[i]); and just after:
>> pthread_join(pthread, NULL);
>>
>> Can we avoid creating this thread?
>> I guess no, because it must spawn on a specific CPU.
>>
>>
> 
> The per thread data seems to break lots of expectations in EAL.
> It all seems to be about capturing the number of cycles on different cores.
> Looks like a mess.

Re: quick thread in DLB2

[Mattias Rönnblom]

On 2023-09-11 16:28, Sevincer, Abdullah wrote:
> Mattias,
> Yes that’s correct.
> 
> 

There is no way to cleaner and more robust way to achieve the same 
result? For example, by accessing /proc, or better, an DPDK abstraction 
of the same.

> -----Original Message-----
> From: Mattias Rönnblom <hofors@lysator.liu.se>
> Sent: Friday, September 8, 2023 12:28 AM
> To: Sevincer, Abdullah <abdullah.sevincer@intel.com>; Stephen Hemminger <stephen@networkplumber.org>; Thomas Monjalon <thomas@monjalon.net>
> Cc: dev@dpdk.org; Tyler Retzlaff <roretzla@linux.microsoft.com>
> Subject: Re: quick thread in DLB2
> 
> On 2023-09-08 00:09, Sevincer, Abdullah wrote:
>> Hi Stephen,
>> It is probing ports for best CPU. Yes it collects cycles. We may rework in the future.
> 
> Best, in what sense? Is this some kind of topology exploration? One DLB port being closer to (cheaper to access for) certain cores?
> 
>> Open to suggestions.
>>
>> -----Original Message-----
>> From: Stephen Hemminger <stephen@networkplumber.org>
>> Sent: Wednesday, September 6, 2023 12:45 PM
>> To: Thomas Monjalon <thomas@monjalon.net>
>> Cc: Sevincer, Abdullah <abdullah.sevincer@intel.com>; dev@dpdk.org;
>> Tyler Retzlaff <roretzla@linux.microsoft.com>
>> Subject: Re: quick thread in DLB2
>>
>> On Fri, 01 Sep 2023 16:08:48 +0200
>> Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>>> Hello Abdullah,
>>>
>>> In the DLB2 code, I see a thread is created for a single operation:
>>> In drivers/event/dlb2/pf/base/dlb2_resource.c
>>> pthread_create(&pthread, NULL, &dlb2_pp_profile_func,
>>> &dlb2_thread_data[i]); and just after:
>>> pthread_join(pthread, NULL);
>>>
>>> Can we avoid creating this thread?
>>> I guess no, because it must spawn on a specific CPU.
>>>
>>>
>>
>> The per thread data seems to break lots of expectations in EAL.
>> It all seems to be about capturing the number of cycles on different cores.
>> Looks like a mess.

RE: quick thread in DLB2

[Honnappa Nagarahalli]

> -----Original Message-----
> From: Mattias Rönnblom <hofors@lysator.liu.se>
> Sent: Wednesday, September 13, 2023 10:48 AM
> To: Sevincer, Abdullah <abdullah.sevincer@intel.com>; Stephen Hemminger
> <stephen@networkplumber.org>; thomas@monjalon.net
> Cc: dev@dpdk.org; Tyler Retzlaff <roretzla@linux.microsoft.com>
> Subject: Re: quick thread in DLB2
> 
> On 2023-09-11 16:28, Sevincer, Abdullah wrote:
> > Mattias,
> > Yes that’s correct.
> >
> >
> 
> There is no way to cleaner and more robust way to achieve the same result?
> For example, by accessing /proc, or better, an DPDK abstraction of the same.
There similar issues in other areas. For ex: the CPUs with large core count have larger interconnect. The SLC to CPU distance starts to matter and the memory latency increases. The distance of the cores on the interconnect also impacts lock behaviors. We probably need a common mechanism/library to export such details. Not sure how much of this would be a security risk.

> 
> > -----Original Message-----
> > From: Mattias Rönnblom <hofors@lysator.liu.se>
> > Sent: Friday, September 8, 2023 12:28 AM
> > To: Sevincer, Abdullah <abdullah.sevincer@intel.com>; Stephen
> > Hemminger <stephen@networkplumber.org>; Thomas Monjalon
> > <thomas@monjalon.net>
> > Cc: dev@dpdk.org; Tyler Retzlaff <roretzla@linux.microsoft.com>
> > Subject: Re: quick thread in DLB2
> >
> > On 2023-09-08 00:09, Sevincer, Abdullah wrote:
> >> Hi Stephen,
> >> It is probing ports for best CPU. Yes it collects cycles. We may rework in the
> future.
> >
> > Best, in what sense? Is this some kind of topology exploration? One DLB
> port being closer to (cheaper to access for) certain cores?
> >
> >> Open to suggestions.
> >>
> >> -----Original Message-----
> >> From: Stephen Hemminger <stephen@networkplumber.org>
> >> Sent: Wednesday, September 6, 2023 12:45 PM
> >> To: Thomas Monjalon <thomas@monjalon.net>
> >> Cc: Sevincer, Abdullah <abdullah.sevincer@intel.com>; dev@dpdk.org;
> >> Tyler Retzlaff <roretzla@linux.microsoft.com>
> >> Subject: Re: quick thread in DLB2
> >>
> >> On Fri, 01 Sep 2023 16:08:48 +0200
> >> Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >>> Hello Abdullah,
> >>>
> >>> In the DLB2 code, I see a thread is created for a single operation:
> >>> In drivers/event/dlb2/pf/base/dlb2_resource.c
> >>> pthread_create(&pthread, NULL, &dlb2_pp_profile_func,
> >>> &dlb2_thread_data[i]); and just after:
> >>> pthread_join(pthread, NULL);
> >>>
> >>> Can we avoid creating this thread?
> >>> I guess no, because it must spawn on a specific CPU.
> >>>
> >>>
> >>
> >> The per thread data seems to break lots of expectations in EAL.
> >> It all seems to be about capturing the number of cycles on different cores.
> >> Looks like a mess.

Re: quick thread in DLB2

[Mattias Rönnblom]

On 2023-09-13 22:56, Honnappa Nagarahalli wrote:
> 
> 
>> -----Original Message-----
>> From: Mattias Rönnblom <hofors@lysator.liu.se>
>> Sent: Wednesday, September 13, 2023 10:48 AM
>> To: Sevincer, Abdullah <abdullah.sevincer@intel.com>; Stephen Hemminger
>> <stephen@networkplumber.org>; thomas@monjalon.net
>> Cc: dev@dpdk.org; Tyler Retzlaff <roretzla@linux.microsoft.com>
>> Subject: Re: quick thread in DLB2
>>
>> On 2023-09-11 16:28, Sevincer, Abdullah wrote:
>>> Mattias,
>>> Yes that’s correct.
>>>
>>>
>>
>> There is no way to cleaner and more robust way to achieve the same result?
>> For example, by accessing /proc, or better, an DPDK abstraction of the same.
> There similar issues in other areas. For ex: the CPUs with large core count have larger interconnect. The SLC to CPU distance starts to matter and the memory latency increases. The distance of the cores on the interconnect also impacts lock behaviors. We probably need a common mechanism/library to export such details.

To make DSW (and other work schedulers) work better on systems with SMT, 
it would be useful to know which lcores are hardware thread siblings.

Topology related to CPU core capacity in heterogeneous system (e.g., 
big.LITTLE) could be used for similar purposes.

The list goes on but one wouldn't need to address all use cases in the 
v1 API.

Something like hwloc(7), but DPDK native.

> Not sure how much of this would be a security risk.
> 

What do have in mind? The DPDK library has no more privileges than the 
application running on top of it. As far as I see, what we are talking 
about here is mere convenience and portability, from a security point of 
view.

>>
>>> -----Original Message-----
>>> From: Mattias Rönnblom <hofors@lysator.liu.se>
>>> Sent: Friday, September 8, 2023 12:28 AM
>>> To: Sevincer, Abdullah <abdullah.sevincer@intel.com>; Stephen
>>> Hemminger <stephen@networkplumber.org>; Thomas Monjalon
>>> <thomas@monjalon.net>
>>> Cc: dev@dpdk.org; Tyler Retzlaff <roretzla@linux.microsoft.com>
>>> Subject: Re: quick thread in DLB2
>>>
>>> On 2023-09-08 00:09, Sevincer, Abdullah wrote:
>>>> Hi Stephen,
>>>> It is probing ports for best CPU. Yes it collects cycles. We may rework in the
>> future.
>>>
>>> Best, in what sense? Is this some kind of topology exploration? One DLB
>> port being closer to (cheaper to access for) certain cores?
>>>
>>>> Open to suggestions.
>>>>
>>>> -----Original Message-----
>>>> From: Stephen Hemminger <stephen@networkplumber.org>
>>>> Sent: Wednesday, September 6, 2023 12:45 PM
>>>> To: Thomas Monjalon <thomas@monjalon.net>
>>>> Cc: Sevincer, Abdullah <abdullah.sevincer@intel.com>; dev@dpdk.org;
>>>> Tyler Retzlaff <roretzla@linux.microsoft.com>
>>>> Subject: Re: quick thread in DLB2
>>>>
>>>> On Fri, 01 Sep 2023 16:08:48 +0200
>>>> Thomas Monjalon <thomas@monjalon.net> wrote:
>>>>
>>>>> Hello Abdullah,
>>>>>
>>>>> In the DLB2 code, I see a thread is created for a single operation:
>>>>> In drivers/event/dlb2/pf/base/dlb2_resource.c
>>>>> pthread_create(&pthread, NULL, &dlb2_pp_profile_func,
>>>>> &dlb2_thread_data[i]); and just after:
>>>>> pthread_join(pthread, NULL);
>>>>>
>>>>> Can we avoid creating this thread?
>>>>> I guess no, because it must spawn on a specific CPU.
>>>>>
>>>>>
>>>>
>>>> The per thread data seems to break lots of expectations in EAL.
>>>> It all seems to be about capturing the number of cycles on different cores.
>>>> Looks like a mess.