[dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[Gregory Etelson]

MP servers distributes RX packets between clients according to
round-robin scheme.

Current implementation always started packets distribution from
the first client. That procedure resulted in uniform distribution
in cases when RX packets number was a multiple of clients number.
However, if RX burst repeatedly returned single
packet, round-robin scheme would not work because all packets
were assigned to the first client only.

The patch does not restart packets distribution from
the first client.
Packets distribution always continues to the next client.

Fixes: af75078fece3 ("first public release")

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Reviewed-by: Dmitry Kozlyuk <dkozlyuk@oss.nvidia.com>
---
 examples/multi_process/client_server_mp/mp_server/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/examples/multi_process/client_server_mp/mp_server/main.c b/examples/multi_process/client_server_mp/mp_server/main.c
index b4761ebc7b..fb441cbbf0 100644
--- a/examples/multi_process/client_server_mp/mp_server/main.c
+++ b/examples/multi_process/client_server_mp/mp_server/main.c
@@ -234,7 +234,7 @@ process_packets(uint32_t port_num __rte_unused,
 		struct rte_mbuf *pkts[], uint16_t rx_count)
 {
 	uint16_t i;
-	uint8_t client = 0;
+	static uint8_t client = 0;
 
 	for (i = 0; i < rx_count; i++) {
 		enqueue_rx_packet(client, pkts[i]);
-- 
2.33.1

Re: [dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[Burakov, Anatoly]

On 26-Oct-21 10:50 AM, Gregory Etelson wrote:
> MP servers distributes RX packets between clients according to
> round-robin scheme.
> 
> Current implementation always started packets distribution from
> the first client. That procedure resulted in uniform distribution
> in cases when RX packets number was a multiple of clients number.
> However, if RX burst repeatedly returned single
> packet, round-robin scheme would not work because all packets
> were assigned to the first client only.
> 
> The patch does not restart packets distribution from
> the first client.
> Packets distribution always continues to the next client.
> 
> Fixes: af75078fece3 ("first public release")
> 
> Signed-off-by: Gregory Etelson <getelson@nvidia.com>
> Reviewed-by: Dmitry Kozlyuk <dkozlyuk@oss.nvidia.com>
> ---
>   examples/multi_process/client_server_mp/mp_server/main.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/examples/multi_process/client_server_mp/mp_server/main.c b/examples/multi_process/client_server_mp/mp_server/main.c
> index b4761ebc7b..fb441cbbf0 100644
> --- a/examples/multi_process/client_server_mp/mp_server/main.c
> +++ b/examples/multi_process/client_server_mp/mp_server/main.c
> @@ -234,7 +234,7 @@ process_packets(uint32_t port_num __rte_unused,
>   		struct rte_mbuf *pkts[], uint16_t rx_count)
>   {
>   	uint16_t i;
> -	uint8_t client = 0;
> +	static uint8_t client = 0;
>   
>   	for (i = 0; i < rx_count; i++) {
>   		enqueue_rx_packet(client, pkts[i]);
> 

Wouldn't that make it global? I don't recall off the top of my head if 
the multiprocess app is intended to have multiple Rx threads, but if you 
did have two forwarding threads, they would effectively both use the 
same `client` value, stepping on top of each other. This should probably 
be per-thread?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[Gregory Etelson]

Hello Anatoly,

..snip..

> b/examples/multi_process/client_server_mp/m
> p_server/main.c
> > @@ -234,7 +234,7 @@
> process_packets(uint32_t port_num
> __rte_unused,
> >               struct rte_mbuf *pkts[], uint16_t
> rx_count)
> >   {
> >       uint16_t i;
> > -     uint8_t client = 0;
> > +     static uint8_t client = 0;
> >
> >       for (i = 0; i < rx_count; i++) {
> >               enqueue_rx_packet(client, pkts[i]);
> >
> 
> Wouldn't that make it global? I don't recall off
> the top of my head if
> the multiprocess app is intended to have
> multiple Rx threads, but if you
> did have two forwarding threads, they would
> effectively both use the
> same `client` value, stepping on top of each
> other. This should probably
> be per-thread?
> 

MP client-server example was not designed as a multi-threaded app.
Server and clients run in a different process and the model allows one server process.
Server allocates a dedicated ring to each client and distributes Rx packets
between rings in round-robin sequence. 
Each ring configured for single producer and single consumer.
Consider an example when server's rte_eth_rx_burst() returns a single packet
on each call. 
Without the patch, the server will ignore all clients with id > 0 and 
assign all Rx packets to rx_ring 0.
Changing process_packets()  `client` variable to static allows unform round-robin
packets distribution between rings.

Regards,
Gregory  

Re: [dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[Burakov, Anatoly]

On 28-Oct-21 4:14 PM, Gregory Etelson wrote:
> Hello Anatoly,
> 
> ..snip..
> 
>> b/examples/multi_process/client_server_mp/m
>> p_server/main.c
>>> @@ -234,7 +234,7 @@
>> process_packets(uint32_t port_num
>> __rte_unused,
>>>                struct rte_mbuf *pkts[], uint16_t
>> rx_count)
>>>    {
>>>        uint16_t i;
>>> -     uint8_t client = 0;
>>> +     static uint8_t client = 0;
>>>
>>>        for (i = 0; i < rx_count; i++) {
>>>                enqueue_rx_packet(client, pkts[i]);
>>>
>>
>> Wouldn't that make it global? I don't recall off
>> the top of my head if
>> the multiprocess app is intended to have
>> multiple Rx threads, but if you
>> did have two forwarding threads, they would
>> effectively both use the
>> same `client` value, stepping on top of each
>> other. This should probably
>> be per-thread?
>>
> 
> MP client-server example was not designed as a multi-threaded app.
> Server and clients run in a different process and the model allows one server process.
> Server allocates a dedicated ring to each client and distributes Rx packets
> between rings in round-robin sequence.
> Each ring configured for single producer and single consumer.
> Consider an example when server's rte_eth_rx_burst() returns a single packet
> on each call.
> Without the patch, the server will ignore all clients with id > 0 and
> assign all Rx packets to rx_ring 0.
> Changing process_packets()  `client` variable to static allows unform round-robin
> packets distribution between rings.
> 
> Regards,
> Gregory
> 

Right, i just checked the code, and the app indeed allows only one 
forwarding thread on the server.

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[Thomas Monjalon]

28/10/2021 17:35, Burakov, Anatoly:
> On 28-Oct-21 4:14 PM, Gregory Etelson wrote:
> >>> -     uint8_t client = 0;
> >>> +     static uint8_t client = 0;
> 
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

checkpatch has a message for you:
ERROR:INITIALISED_STATIC: do not initialise statics to 0

Re: [dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[Gregory Etelson]

Hello Thomas,

> 
> 28/10/2021 17:35, Burakov, Anatoly:
> > On 28-Oct-21 4:14 PM, Gregory Etelson wrote:
> > >>> -     uint8_t client = 0;
> > >>> +     static uint8_t client = 0;
> >
> > Acked-by: Anatoly Burakov
> <anatoly.burakov@intel.com>
> 
> checkpatch has a message for you:
> ERROR:INITIALISED_STATIC: do not initialise
> statics to 0
> 

Turning the `client` variable to static ensured that the next time
the function will be called it will proceed iterating clients instead of
starting a loop from the beginning - that's the main idea of that patch.
The variable must be initialized to 0 because the application model 
requires at least a single client with index 0.
ANSI C allows static variables initialization to any valid value.
Do you know why the checkpatch utility denied such initialization ?

Regards,
Gregory

 

Re: [dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[Thomas Monjalon]

09/11/2021 07:42, Gregory Etelson:
> Hello Thomas,
> 
> > 
> > 28/10/2021 17:35, Burakov, Anatoly:
> > > On 28-Oct-21 4:14 PM, Gregory Etelson wrote:
> > > >>> -     uint8_t client = 0;
> > > >>> +     static uint8_t client = 0;
> > >
> > > Acked-by: Anatoly Burakov
> > <anatoly.burakov@intel.com>
> > 
> > checkpatch has a message for you:
> > ERROR:INITIALISED_STATIC: do not initialise
> > statics to 0
> > 
> 
> Turning the `client` variable to static ensured that the next time
> the function will be called it will proceed iterating clients instead of
> starting a loop from the beginning - that's the main idea of that patch.
> The variable must be initialized to 0 because the application model 
> requires at least a single client with index 0.
> ANSI C allows static variables initialization to any valid value.
> Do you know why the checkpatch utility denied such initialization ?

ANSI C makes static variables iniatilized to 0 by default.

Re: [dpdk-dev] [PATCH] examples/multi_process: fix RX packets distribution

[Gregory Etelson]

Hello Thomas,


> > > 28/10/2021 17:35, Burakov, Anatoly:
> > > > On 28-Oct-21 4:14 PM, Gregory Etelson
> wrote:
> > > > >>> -     uint8_t client = 0;
> > > > >>> +     static uint8_t client = 0;
> > > >
> > > > Acked-by: Anatoly Burakov
> > > <anatoly.burakov@intel.com>
> > >
> > > checkpatch has a message for you:
> > > ERROR:INITIALISED_STATIC: do not initialise
> > > statics to 0
> > >
> >
> > Turning the `client` variable to static ensured
> that the next time
> > the function will be called it will proceed
> iterating clients instead of
> > starting a loop from the beginning - that's the
> main idea of that patch.
> > The variable must be initialized to 0 because
> the application model
> > requires at least a single client with index 0.
> > ANSI C allows static variables initialization to
> any valid value.
> > Do you know why the checkpatch utility denied
> such initialization ?
> 
> ANSI C makes static variables iniatilized to 0 by
> default.
> 


I'll post updated patch

Regards,
Gregory

[dpdk-dev] [PATCH v2] examples/multi_proces: fix Rx packets distribution

[Gregory Etelson]

MP servers distributes Rx packets between clients according to
round-robin scheme.

Current implementation always started packets distribution from
the first client. That procedure resulted in uniform distribution
in cases when Rx packets number was around clients number
multiplication. However, if RX burst repeatedly returned single
packet, round-robin scheme would not work because all packets
were assigned to the first client only.

The patch does not restart packets distribution from
the first client.
Packets distribution always continues to the next client.

Cc: stable@dpdk.org

Fixes: af75078fece3 ("first public release")

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
v2: Remove explisit static variable initialization.
---
 examples/multi_process/client_server_mp/mp_server/main.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/examples/multi_process/client_server_mp/mp_server/main.c b/examples/multi_process/client_server_mp/mp_server/main.c
index b4761ebc7b..31e7e76706 100644
--- a/examples/multi_process/client_server_mp/mp_server/main.c
+++ b/examples/multi_process/client_server_mp/mp_server/main.c
@@ -234,7 +234,12 @@ process_packets(uint32_t port_num __rte_unused,
 		struct rte_mbuf *pkts[], uint16_t rx_count)
 {
 	uint16_t i;
-	uint8_t client = 0;
+	/*
+	 * C99: All objects with static storage duration
+	 * shall be initialized (set to their initial values) before
+	 * program startup.
+	 */
+	static uint8_t client;
 
 	for (i = 0; i < rx_count; i++) {
 		enqueue_rx_packet(client, pkts[i]);
-- 
2.33.1

Re: [dpdk-dev] [PATCH v2] examples/multi_proces: fix Rx packets distribution

[Thomas Monjalon]

09/11/2021 10:58, Gregory Etelson:
> -	uint8_t client = 0;
> +	/*
> +	 * C99: All objects with static storage duration
> +	 * shall be initialized (set to their initial values) before
> +	 * program startup.
> +	 */

Why adding this comment?

> +	static uint8_t client;

Re: [dpdk-dev] [PATCH v2] examples/multi_proces: fix Rx packets distribution

[Gregory Etelson]

Hello Thomas,

> 09/11/2021 10:58, Gregory Etelson:
> > -     uint8_t client = 0;
> > +     /*
> > +      * C99: All objects with static storage
> duration
> > +      * shall be initialized (set to their initial
> values) before
> > +      * program startup.
> > +      */
> 
> Why adding this comment?
> 
> > +     static uint8_t client;
> 
> 

C99 optimization that was used here is not obvious.
The patch relies on client=0 initialization.
I added the comment to clarify why the client was not initialized.

Re: [dpdk-dev] [PATCH v2] examples/multi_proces: fix Rx packets distribution

[Thomas Monjalon]

09/11/2021 12:49, Gregory Etelson:
> Hello Thomas,
> 
> > 09/11/2021 10:58, Gregory Etelson:
> > > -     uint8_t client = 0;
> > > +     /*
> > > +      * C99: All objects with static storage
> > duration
> > > +      * shall be initialized (set to their initial
> > values) before
> > > +      * program startup.
> > > +      */
> > 
> > Why adding this comment?
> > 
> > > +     static uint8_t client;
> > 
> > 
> 
> C99 optimization that was used here is not obvious.
> The patch relies on client=0 initialization.
> I added the comment to clarify why the client was not initialized.

I think it is the C standard, not only C99.
As far as I know, having static as 0 is obvious for a lot of people.

[PATCH v3] examples/multi_proces: fix Rx packets distribution

[Gregory Etelson]

MP servers distributes Rx packets between clients according to
round-robin scheme.

Current implementation always started packets distribution from
the first client. That procedure resulted in uniform distribution
in cases when Rx packets number was around clients number
multiplication. However, if RX burst repeatedly returned single
packet, round-robin scheme would not work because all packets
were assigned to the first client only.

The patch does not restart packets distribution from
the first client.
Packets distribution always continues to the next client.

Cc: stable@dpdk.org

Fixes: af75078fece3 ("first public release")

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
v2: Remove explisit static variable initialization.
v3: Remove comment.
---
 examples/multi_process/client_server_mp/mp_server/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/examples/multi_process/client_server_mp/mp_server/main.c b/examples/multi_process/client_server_mp/mp_server/main.c
index b4761ebc7b..f54bb8b75a 100644
--- a/examples/multi_process/client_server_mp/mp_server/main.c
+++ b/examples/multi_process/client_server_mp/mp_server/main.c
@@ -234,7 +234,7 @@ process_packets(uint32_t port_num __rte_unused,
 		struct rte_mbuf *pkts[], uint16_t rx_count)
 {
 	uint16_t i;
-	uint8_t client = 0;
+	static uint8_t client;
 
 	for (i = 0; i < rx_count; i++) {
 		enqueue_rx_packet(client, pkts[i]);
-- 
2.33.1

[PATCH v4] examples/multi_process: fix Rx packets distribution

[Gregory Etelson]

MP servers distributes Rx packets between clients according to
round-robin scheme.

Current implementation always started packets distribution from
the first client. That procedure resulted in uniform distribution
in cases when Rx packets number was around clients number
multiplication. However, if RX burst repeatedly returned single
packet, round-robin scheme would not work because all packets
were assigned to the first client only.

The patch does not restart packets distribution from
the first client.
Packets distribution always continues to the next client.

Cc: stable@dpdk.org

Fixes: af75078fece3 ("first public release")

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
v2: Remove explisit static variable initialization.
v3: Remove comment.
v4: Spell check.
---
 examples/multi_process/client_server_mp/mp_server/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/examples/multi_process/client_server_mp/mp_server/main.c b/examples/multi_process/client_server_mp/mp_server/main.c
index b4761ebc7b..f54bb8b75a 100644
--- a/examples/multi_process/client_server_mp/mp_server/main.c
+++ b/examples/multi_process/client_server_mp/mp_server/main.c
@@ -234,7 +234,7 @@ process_packets(uint32_t port_num __rte_unused,
 		struct rte_mbuf *pkts[], uint16_t rx_count)
 {
 	uint16_t i;
-	uint8_t client = 0;
+	static uint8_t client;
 
 	for (i = 0; i < rx_count; i++) {
 		enqueue_rx_packet(client, pkts[i]);
-- 
2.33.1

Re: [PATCH v4] examples/multi_process: fix Rx packets distribution

[David Marchand]

On Wed, Nov 10, 2021 at 5:58 PM Gregory Etelson <getelson@nvidia.com> wrote:
>
> MP servers distributes Rx packets between clients according to
> round-robin scheme.
>
> Current implementation always started packets distribution from
> the first client. That procedure resulted in uniform distribution
> in cases when Rx packets number was around clients number
> multiplication. However, if RX burst repeatedly returned single
> packet, round-robin scheme would not work because all packets
> were assigned to the first client only.
>
> The patch does not restart packets distribution from
> the first client.
> Packets distribution always continues to the next client.
>
> Fixes: af75078fece3 ("first public release")
> Cc: stable@dpdk.org
>
> Signed-off-by: Gregory Etelson <getelson@nvidia.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

Applied, thanks.


-- 
David Marchand