[dpdk-dev] 回复: [EXT] [PATCH v3] examples/eventdev: refactor ethdev port stop

[Feifei Wang <Feifei.Wang2@arm.com>]

> -----邮件原件-----
> 发件人: Feifei Wang
> 发送时间: 2021年1月14日 14:24
> 收件人: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Harry
> van Haaren <harry.van.haaren@intel.com>; Nikhil Rao
> <nikhil.rao@intel.com>; Pavan Nikhilesh
> <pbhagavatula@caviumnetworks.com>
> 抄送: dev@dpdk.org; jerinj@marvell.com; nd <nd@arm.com>;
> stable@dpdk.org; Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> 主题: 回复: [EXT] [PATCH v3] examples/eventdev: refactor ethdev port stop
> 
> > -----邮件原件-----
> > 发件人: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>
> > 发送时间: 2021年1月5日 18:09
> > 收件人: Feifei Wang <Feifei.Wang2@arm.com>; Harry van Haaren
> > <harry.van.haaren@intel.com>; Nikhil Rao <nikhil.rao@intel.com>; Pavan
> > Nikhilesh <pbhagavatula@caviumnetworks.com>
> > 抄送: dev@dpdk.org; jerinj@marvell.com; nd <nd@arm.com>;
> > stable@dpdk.org; Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa
> > Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > 主题: RE: [EXT] [PATCH v3] examples/eventdev: refactor ethdev port stop
> >
> Hi, Pavan
> 
> > Hi Feifei,
> >
> > >Move eth stop code from "signal_handler" function to the end of "main"
> > >function. There are two reasons for this:
> > >
> > >First, this improves code maintenance and makes code look simple and
> > >clear. Based on this change, after receiving the interrupt signal,
> > >"fdata->done" is set as 1. Then the main thread will wait all worker
> > >lcores to jump out of the loop. Finally, the main thread will stop
> > >and then close eth dev port.
> > >
> > >Second, for older version, the main thread first stops eth dev port
> > >and then waits the end of worker lcore. This may cause errors because
> > >it may stop the eth dev port which worker lcores are using. This
> > >moving change can fix this by waiting all worker threads to exit and
> > >then stop the eth dev port.
> >
> > Apologies for the delayed reply,
> >
> > In case of event dev the workers don't interact with eth device
> > directly, Instead eth device "injects" packets into event device and
> > event device is responsible for scheduling them to the workers.
> >
> > If the producer is not stopped i.e. in this case eth device then the
> > worker threads might never exit and the main core would wait
> > indefinitely for workers to exit. This will be predominantly seen in
> > cases where there are only a few flows and large number of workers
> > causing a lot of intra thread dependency.
> 
> For the case that the event device scheduling packets to the workers.
> Though the producer is not stopped (eth device), when the main core
> receive the interrupt signal, the "fdata->done"
> will be set as 1. Then all the workers load the value 1 of "fdata->done" and
> jump out of the loop to finish their thread.
> 

And I also test the case you said above with this patch, when I send interrupt signal(ctrl + c), IXIA
is running and eth device is also working, then the program can exit normally.

HW:
Traffic generator: IXIA 
Nics: ixgbe 82599ES 10-Gigabit
Architecture: aarch64
CPU: Cortex-A72

With this patch:
______________________________________________________________________________________________
Command: sudo ./dpdk-eventdev_pipeline -l 0-2,8-11 --vdev=event_sw0 -- -r2 -t2 -e4 -w F00 -s4 -n0 -c32 -W1000 -D
.........
Config:
        ports: 2
        workers: 4
        packets: 9223372036854775807
        Queue-prio: 0
        qid0 type: atomic
        Cores available: 7
        Cores used: 6
        Eventdev 0: event_sw
  Stages:
        Stage 0, Type Atomic    Priority = 128
        Stage 1, Type Atomic    Priority = 128
        Stage 2, Type Atomic    Priority = 128
        Stage 3, Type Atomic    Priority = 128

Port 0 modified RSS hash function based on hardware support,requested:0x3afbc configured:0x38d34
Port 0 MAC: 90 e2 ba 56 ed 6c
Port 1 modified RSS hash function based on hardware support,requested:0x3afbc configured:0x38d34
Port 1 MAC: 90 e2 ba 56 ed 6d
[dump_core_info()] lcore 1 executing NIC Rx
[dump_core_info()] lcore 1 executing NIC Tx
[dump_core_info()] lcore 2 executing scheduler
[dump_core_info()] lcore 8 executing worker, using eventdev port 0
[dump_core_info()] lcore 9 executing worker, using eventdev port 1
[dump_core_info()] lcore 10 executing worker, using eventdev port 2
[dump_core_info()] lcore 11 executing worker, using eventdev port 3


Command: ctrl + c
.........
  worker 1 thread done. RX=0 TX=0
  worker 2 thread done. RX=0 TX=0
  worker 11 thread done. RX=1844023 TX=1844023
  worker 9 thread done. RX=1844888 TX=1844888
  worker 8 thread done. RX=1845490 TX=1845490
  worker 10 thread done. RX=1844903 TX=1844903

Port Workload distribution:
worker 0 :      25.0 % (1845474 pkts)
worker 1 :      25.0 % (1844872 pkts)
worker 2 :      25.0 % (1844887 pkts)
worker 3 :      25.0 % (1844007 pkts)
Invalid port_id=0
Invalid port_id=1
______________________________________________________________________________________________

> Best Regards
> Feifei
> >
> > Regards,
> > Pavan.
> >
> > >
> > >In the meanwhile, remove wmb in signal_handler.
> > >
> > >This is because when the main lcore receive the stop signal, it
> > >stores
> > >1 into fdata->done. And then the worker lcores load "fdata->done" and
> > >jump out of the loop to stop running. Nothing should be stored after
> > >updating
> > >fdata->done, so the wmb is unnecessary.
> > >
> > >Fixes: 085edac2ca38 ("examples/eventdev_pipeline: support Tx
> > >adapter")
> > >Cc: pbhagavatula@marvell.com
> > >Cc: stable@dpdk.org
> > >
> > >Suggested-by: Ruifeng Wang <ruifeng.wang@arm.com>
> > >Signed-off-by: Feifei Wang <feifei.wang2@arm.com>
> > >Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> > >Reviewed-by: Honnappa Nagarahalli
> > ><honnappa.nagarahalli@arm.com>
> > >Acked-by: Harry van Haaren <harry.van.haaren@intel.com>
> > >---
> > >
> > >v2:
> > >1. Delete unused variable to fix build error 2. Reduce commit message
> > >length to fix coding style issues
> > >
> > >v3:
> > >1. Title improvement (Van Haaren)
> > >
> > > examples/eventdev_pipeline/main.c | 16 ++++------------
> > > 1 file changed, 4 insertions(+), 12 deletions(-)
> > >
> > >diff --git a/examples/eventdev_pipeline/main.c
> > >b/examples/eventdev_pipeline/main.c
> > >index 823f8b51c..fdbaf667b 100644
> > >--- a/examples/eventdev_pipeline/main.c
> > >+++ b/examples/eventdev_pipeline/main.c
> > >@@ -280,7 +280,6 @@ static void
> > > signal_handler(int signum)
> > > {
> > > 	static uint8_t once;
> > >-	uint16_t portid;
> > >
> > > 	if (fdata->done)
> > > 		rte_exit(1, "Exiting on signal %d\n", signum); @@ -291,17
> > +290,6 @@
> > >signal_handler(int signum)
> > > 			rte_event_dev_dump(0, stdout);
> > > 		once = 1;
> > > 		fdata->done = 1;
> > >-		rte_smp_wmb();
> > >-
> > >-		RTE_ETH_FOREACH_DEV(portid) {
> > >-			rte_event_eth_rx_adapter_stop(portid);
> > >-			rte_event_eth_tx_adapter_stop(portid);
> > >-			if (rte_eth_dev_stop(portid) < 0)
> > >-				printf("Failed to stop port %u", portid);
> > >-		}
> > >-
> > >-		rte_eal_mp_wait_lcore();
> > >-
> > > 	}
> > > 	if (signum == SIGTSTP)
> > > 		rte_event_dev_dump(0, stdout);
> > >@@ -465,6 +453,10 @@ main(int argc, char **argv)
> > > 	}
> > >
> > > 	RTE_ETH_FOREACH_DEV(portid) {
> > >+		rte_event_eth_rx_adapter_stop(portid);
> > >+		rte_event_eth_tx_adapter_stop(portid);
> > >+		if (rte_eth_dev_stop(portid) < 0)
> > >+			printf("Failed to stop port %u", portid);
> > > 		rte_eth_dev_close(portid);
> > > 	}
> > >
> > >--
> > >2.25.1