[dpdk-dev] Physical core vs. hyper threaded core

[dpdk-dev] Physical core vs. hyper threaded core

[Jane Shen]

Hi,

I understand that DPDK should use the physical core. But here is what we tested:

-          Enable HT

-          Assign 8 cores of the CPU (an 8-core Sandybridge CPU) to DPDK.

Surprisingly enough, we noticed that the remaining 8 cores (b/c there are total of 16 cores after HT) can still handle other Linux processes which are SIP based signaling transactions.

Anybody can shed some light on how this worked? Is there anybody tried similar thing? What has been your experience?

Thanks,
-Jane

Re: [dpdk-dev] Physical core vs. hyper threaded core

[Jayakumar, Muthurajan]

Jane, 
Great. You are correct. Have tried enabling hyperthreading and it works. 

For example, if we want to have the functionality partitioning such that Rx + Packet Processing + Tx = all of these three functions can be done in  2 cores  - By positioning Rx in one lcore and by positioning Packet processing and Tx in the sibling hyperthread lcore of the same physical core, you get tight coupling because L1 cache and L2 cache are shared between the hyperthreaded cores belonging to same physical core.   

Curious to know - in your configuration, the SIP based signaling threads - 

Option A) are they sharing sibling of DPDK threads?

Option B) Or all DPDK threads are tightly coupled with sibling threads and SIP based signaling threads are on separate cores?

If  it is Option B) more tight coupling within the DPDK threads and less interference from signaling threads. 

Thanks, 


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jane Shen
Sent: Sunday, March 02, 2014 4:56 PM
To: dev@dpdk.org
Subject: [dpdk-dev] Physical core vs. hyper threaded core

Hi,

I understand that DPDK should use the physical core. But here is what we tested:

-          Enable HT

-          Assign 8 cores of the CPU (an 8-core Sandybridge CPU) to DPDK.

Surprisingly enough, we noticed that the remaining 8 cores (b/c there are total of 16 cores after HT) can still handle other Linux processes which are SIP based signaling transactions.

Anybody can shed some light on how this worked? Is there anybody tried similar thing? What has been your experience?

Thanks,
-Jane

Re: [dpdk-dev] Physical core vs. hyper threaded core

[Jane Shen]

Thanks Muthurajan.

We were testing with core 0-7 to DPDK and 8-15 to Linux SIP processes. The core numbers are based on the Linux /etc/cpuinfo. These processes don't have direct coupling with DPDK. They may share some memory with the DPDK via IPC.

What was the reason that it is recommended to use the physical core only for DPDK? Based on your comment below, sounds like that restriction should be removed.


Thanks,
-Jane


-----Original Message-----
From: Jayakumar, Muthurajan [mailto:muthurajan.jayakumar@intel.com] 
Sent: Sunday, March 02, 2014 7:49 PM
To: Jane Shen; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Jane, 
Great. You are correct. Have tried enabling hyperthreading and it works. 

For example, if we want to have the functionality partitioning such that Rx + Packet Processing + Tx = all of these three functions can be done in  2 cores  - By positioning Rx in one lcore and by positioning Packet processing and Tx in the sibling hyperthread lcore of the same physical core, you get tight coupling because L1 cache and L2 cache are shared between the hyperthreaded cores belonging to same physical core.   

Curious to know - in your configuration, the SIP based signaling threads - 

Option A) are they sharing sibling of DPDK threads?

Option B) Or all DPDK threads are tightly coupled with sibling threads and SIP based signaling threads are on separate cores?

If  it is Option B) more tight coupling within the DPDK threads and less interference from signaling threads. 

Thanks, 


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jane Shen
Sent: Sunday, March 02, 2014 4:56 PM
To: dev@dpdk.org
Subject: [dpdk-dev] Physical core vs. hyper threaded core

Hi,

I understand that DPDK should use the physical core. But here is what we tested:

-          Enable HT

-          Assign 8 cores of the CPU (an 8-core Sandybridge CPU) to DPDK.

Surprisingly enough, we noticed that the remaining 8 cores (b/c there are total of 16 cores after HT) can still handle other Linux processes which are SIP based signaling transactions.

Anybody can shed some light on how this worked? Is there anybody tried similar thing? What has been your experience?

Thanks,
-Jane

Re: [dpdk-dev] Physical core vs. hyper threaded core

[Jayakumar, Muthurajan]

Thanks Jane. Great. So interference with DPDK. Perfect. 

The reason for using physical core is, you get more resources compared to using in hyperthreaded mode this is because there are some resources statically partitioned and so when one disables hyperthreaded mode and use physical core, then all the resources that are in the core are available.

So, when all the work - Rx, Processing, Tx are to be done by one core itself, then using full physical core without hyperthreading may be the better choice. 
Whereas, in case one application demands pipelined implementation and you are choosing 2 lcores to implement, then one may be better off enabling hyperthreading and pipelining the functionality between the siblings because in this case inter-lcore communication cost is more important.

Thanks,  

-----Original Message-----
From: Jane Shen [mailto:jshen@mavenir.com] 
Sent: Sunday, March 02, 2014 6:14 PM
To: Jayakumar, Muthurajan; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Thanks Muthurajan.

We were testing with core 0-7 to DPDK and 8-15 to Linux SIP processes. The core numbers are based on the Linux /etc/cpuinfo. These processes don't have direct coupling with DPDK. They may share some memory with the DPDK via IPC.

What was the reason that it is recommended to use the physical core only for DPDK? Based on your comment below, sounds like that restriction should be removed.


Thanks,
-Jane


-----Original Message-----
From: Jayakumar, Muthurajan [mailto:muthurajan.jayakumar@intel.com] 
Sent: Sunday, March 02, 2014 7:49 PM
To: Jane Shen; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Jane, 
Great. You are correct. Have tried enabling hyperthreading and it works. 

For example, if we want to have the functionality partitioning such that Rx + Packet Processing + Tx = all of these three functions can be done in  2 cores  - By positioning Rx in one lcore and by positioning Packet processing and Tx in the sibling hyperthread lcore of the same physical core, you get tight coupling because L1 cache and L2 cache are shared between the hyperthreaded cores belonging to same physical core.   

Curious to know - in your configuration, the SIP based signaling threads - 

Option A) are they sharing sibling of DPDK threads?

Option B) Or all DPDK threads are tightly coupled with sibling threads and SIP based signaling threads are on separate cores?

If  it is Option B) more tight coupling within the DPDK threads and less interference from signaling threads. 

Thanks, 


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jane Shen
Sent: Sunday, March 02, 2014 4:56 PM
To: dev@dpdk.org
Subject: [dpdk-dev] Physical core vs. hyper threaded core

Hi,

I understand that DPDK should use the physical core. But here is what we tested:

-          Enable HT

-          Assign 8 cores of the CPU (an 8-core Sandybridge CPU) to DPDK.

Surprisingly enough, we noticed that the remaining 8 cores (b/c there are total of 16 cores after HT) can still handle other Linux processes which are SIP based signaling transactions.

Anybody can shed some light on how this worked? Is there anybody tried similar thing? What has been your experience?

Thanks,
-Jane

Re: [dpdk-dev] Physical core vs. hyper threaded core

[Jayakumar, Muthurajan]

In your scenario, NO interference with DPDK. Perfect.

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jayakumar, Muthurajan
Sent: Sunday, March 02, 2014 6:23 PM
To: Jane Shen; dev@dpdk.org
Subject: Re: [dpdk-dev] Physical core vs. hyper threaded core

Thanks Jane. Great. So interference with DPDK. Perfect. 

The reason for using physical core is, you get more resources compared to using in hyperthreaded mode this is because there are some resources statically partitioned and so when one disables hyperthreaded mode and use physical core, then all the resources that are in the core are available.

So, when all the work - Rx, Processing, Tx are to be done by one core itself, then using full physical core without hyperthreading may be the better choice. 
Whereas, in case one application demands pipelined implementation and you are choosing 2 lcores to implement, then one may be better off enabling hyperthreading and pipelining the functionality between the siblings because in this case inter-lcore communication cost is more important.

Thanks,  

-----Original Message-----
From: Jane Shen [mailto:jshen@mavenir.com] 
Sent: Sunday, March 02, 2014 6:14 PM
To: Jayakumar, Muthurajan; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Thanks Muthurajan.

We were testing with core 0-7 to DPDK and 8-15 to Linux SIP processes. The core numbers are based on the Linux /etc/cpuinfo. These processes don't have direct coupling with DPDK. They may share some memory with the DPDK via IPC.

What was the reason that it is recommended to use the physical core only for DPDK? Based on your comment below, sounds like that restriction should be removed.


Thanks,
-Jane


-----Original Message-----
From: Jayakumar, Muthurajan [mailto:muthurajan.jayakumar@intel.com] 
Sent: Sunday, March 02, 2014 7:49 PM
To: Jane Shen; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Jane, 
Great. You are correct. Have tried enabling hyperthreading and it works. 

For example, if we want to have the functionality partitioning such that Rx + Packet Processing + Tx = all of these three functions can be done in  2 cores  - By positioning Rx in one lcore and by positioning Packet processing and Tx in the sibling hyperthread lcore of the same physical core, you get tight coupling because L1 cache and L2 cache are shared between the hyperthreaded cores belonging to same physical core.   

Curious to know - in your configuration, the SIP based signaling threads - 

Option A) are they sharing sibling of DPDK threads?

Option B) Or all DPDK threads are tightly coupled with sibling threads and SIP based signaling threads are on separate cores?

If  it is Option B) more tight coupling within the DPDK threads and less interference from signaling threads. 

Thanks, 


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jane Shen
Sent: Sunday, March 02, 2014 4:56 PM
To: dev@dpdk.org
Subject: [dpdk-dev] Physical core vs. hyper threaded core

Hi,

I understand that DPDK should use the physical core. But here is what we tested:

-          Enable HT

-          Assign 8 cores of the CPU (an 8-core Sandybridge CPU) to DPDK.

Surprisingly enough, we noticed that the remaining 8 cores (b/c there are total of 16 cores after HT) can still handle other Linux processes which are SIP based signaling transactions.

Anybody can shed some light on how this worked? Is there anybody tried similar thing? What has been your experience?

Thanks,
-Jane

Re: [dpdk-dev] Physical core vs. hyper threaded core

[Jane Shen]

Thanks Muthurajan.

" because there are some resources statically partitioned"

What are these resources and in general what types of operations from the software will be impacted? Like memory access, IO access etc.?

Under what condition would sharing sibling lcores for 2 DPDK threads would be recommended (or at least worth trying)? 

Thanks,
-Jane

-----Original Message-----
From: Jayakumar, Muthurajan [mailto:muthurajan.jayakumar@intel.com] 
Sent: Sunday, March 02, 2014 8:54 PM
To: Jayakumar, Muthurajan; Jane Shen; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

In your scenario, NO interference with DPDK. Perfect.

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jayakumar, Muthurajan
Sent: Sunday, March 02, 2014 6:23 PM
To: Jane Shen; dev@dpdk.org
Subject: Re: [dpdk-dev] Physical core vs. hyper threaded core

Thanks Jane. Great. So interference with DPDK. Perfect. 

The reason for using physical core is, you get more resources compared to using in hyperthreaded mode this is because there are some resources statically partitioned and so when one disables hyperthreaded mode and use physical core, then all the resources that are in the core are available.

So, when all the work - Rx, Processing, Tx are to be done by one core itself, then using full physical core without hyperthreading may be the better choice. 
Whereas, in case one application demands pipelined implementation and you are choosing 2 lcores to implement, then one may be better off enabling hyperthreading and pipelining the functionality between the siblings because in this case inter-lcore communication cost is more important.

Thanks,  

-----Original Message-----
From: Jane Shen [mailto:jshen@mavenir.com] 
Sent: Sunday, March 02, 2014 6:14 PM
To: Jayakumar, Muthurajan; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Thanks Muthurajan.

We were testing with core 0-7 to DPDK and 8-15 to Linux SIP processes. The core numbers are based on the Linux /etc/cpuinfo. These processes don't have direct coupling with DPDK. They may share some memory with the DPDK via IPC.

What was the reason that it is recommended to use the physical core only for DPDK? Based on your comment below, sounds like that restriction should be removed.


Thanks,
-Jane


-----Original Message-----
From: Jayakumar, Muthurajan [mailto:muthurajan.jayakumar@intel.com] 
Sent: Sunday, March 02, 2014 7:49 PM
To: Jane Shen; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Jane, 
Great. You are correct. Have tried enabling hyperthreading and it works. 

For example, if we want to have the functionality partitioning such that Rx + Packet Processing + Tx = all of these three functions can be done in  2 cores  - By positioning Rx in one lcore and by positioning Packet processing and Tx in the sibling hyperthread lcore of the same physical core, you get tight coupling because L1 cache and L2 cache are shared between the hyperthreaded cores belonging to same physical core.   

Curious to know - in your configuration, the SIP based signaling threads - 

Option A) are they sharing sibling of DPDK threads?

Option B) Or all DPDK threads are tightly coupled with sibling threads and SIP based signaling threads are on separate cores?

If  it is Option B) more tight coupling within the DPDK threads and less interference from signaling threads. 

Thanks, 


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jane Shen
Sent: Sunday, March 02, 2014 4:56 PM
To: dev@dpdk.org
Subject: [dpdk-dev] Physical core vs. hyper threaded core

Hi,

I understand that DPDK should use the physical core. But here is what we tested:

-          Enable HT

-          Assign 8 cores of the CPU (an 8-core Sandybridge CPU) to DPDK.

Surprisingly enough, we noticed that the remaining 8 cores (b/c there are total of 16 cores after HT) can still handle other Linux processes which are SIP based signaling transactions.

Anybody can shed some light on how this worked? Is there anybody tried similar thing? What has been your experience?

Thanks,
-Jane

Re: [dpdk-dev] Physical core vs. hyper threaded core

[Jayakumar, Muthurajan]

Thanks Jane. 

Statically allocated are resources like load buffer, store buffer, reorder buffer and small page ITB. 
In addition, there are other resources that are replicated, competitively shared.

Below are couple of links that you will find useful. 

http://software.intel.com/en-us/articles/performance-insights-to-intel-hyper-threading-technology

http://software.intel.com/en-us/articles/avoid-resource-contention-on-hyper-threading-technology-enabled-systems

Thanks,

-----Original Message-----
From: Jane Shen [mailto:jshen@mavenir.com] 
Sent: Monday, March 03, 2014 11:50 AM
To: Jayakumar, Muthurajan; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Thanks Muthurajan.

" because there are some resources statically partitioned"

What are these resources and in general what types of operations from the software will be impacted? Like memory access, IO access etc.?

Under what condition would sharing sibling lcores for 2 DPDK threads would be recommended (or at least worth trying)? 

Thanks,
-Jane

-----Original Message-----
From: Jayakumar, Muthurajan [mailto:muthurajan.jayakumar@intel.com] 
Sent: Sunday, March 02, 2014 8:54 PM
To: Jayakumar, Muthurajan; Jane Shen; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

In your scenario, NO interference with DPDK. Perfect.

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jayakumar, Muthurajan
Sent: Sunday, March 02, 2014 6:23 PM
To: Jane Shen; dev@dpdk.org
Subject: Re: [dpdk-dev] Physical core vs. hyper threaded core

Thanks Jane. Great. So interference with DPDK. Perfect. 

The reason for using physical core is, you get more resources compared to using in hyperthreaded mode this is because there are some resources statically partitioned and so when one disables hyperthreaded mode and use physical core, then all the resources that are in the core are available.

So, when all the work - Rx, Processing, Tx are to be done by one core itself, then using full physical core without hyperthreading may be the better choice. 
Whereas, in case one application demands pipelined implementation and you are choosing 2 lcores to implement, then one may be better off enabling hyperthreading and pipelining the functionality between the siblings because in this case inter-lcore communication cost is more important.

Thanks,  

-----Original Message-----
From: Jane Shen [mailto:jshen@mavenir.com] 
Sent: Sunday, March 02, 2014 6:14 PM
To: Jayakumar, Muthurajan; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Thanks Muthurajan.

We were testing with core 0-7 to DPDK and 8-15 to Linux SIP processes. The core numbers are based on the Linux /etc/cpuinfo. These processes don't have direct coupling with DPDK. They may share some memory with the DPDK via IPC.

What was the reason that it is recommended to use the physical core only for DPDK? Based on your comment below, sounds like that restriction should be removed.


Thanks,
-Jane


-----Original Message-----
From: Jayakumar, Muthurajan [mailto:muthurajan.jayakumar@intel.com] 
Sent: Sunday, March 02, 2014 7:49 PM
To: Jane Shen; dev@dpdk.org
Subject: RE: Physical core vs. hyper threaded core

Jane, 
Great. You are correct. Have tried enabling hyperthreading and it works. 

For example, if we want to have the functionality partitioning such that Rx + Packet Processing + Tx = all of these three functions can be done in  2 cores  - By positioning Rx in one lcore and by positioning Packet processing and Tx in the sibling hyperthread lcore of the same physical core, you get tight coupling because L1 cache and L2 cache are shared between the hyperthreaded cores belonging to same physical core.   

Curious to know - in your configuration, the SIP based signaling threads - 

Option A) are they sharing sibling of DPDK threads?

Option B) Or all DPDK threads are tightly coupled with sibling threads and SIP based signaling threads are on separate cores?

If  it is Option B) more tight coupling within the DPDK threads and less interference from signaling threads. 

Thanks, 


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jane Shen
Sent: Sunday, March 02, 2014 4:56 PM
To: dev@dpdk.org
Subject: [dpdk-dev] Physical core vs. hyper threaded core

Hi,

I understand that DPDK should use the physical core. But here is what we tested:

-          Enable HT

-          Assign 8 cores of the CPU (an 8-core Sandybridge CPU) to DPDK.

Surprisingly enough, we noticed that the remaining 8 cores (b/c there are total of 16 cores after HT) can still handle other Linux processes which are SIP based signaling transactions.

Anybody can shed some light on how this worked? Is there anybody tried similar thing? What has been your experience?

Thanks,
-Jane