Re: [dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-secgw

[Anoob Joseph <anoobj@marvell.com>]

Hi Konstantin,

Thanks for the review. Please see inline.

Thanks,
Anoob

> -----Original Message-----
> From: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Sent: Wednesday, October 16, 2019 6:33 PM
> To: Anoob Joseph <anoobj@marvell.com>; Akhil Goyal
> <akhil.goyal@nxp.com>; Nicolau, Radu <radu.nicolau@intel.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Narayana Prasad Raju Athreya
> <pathreya@marvell.com>; Lukas Bartosik <lbartosik@marvell.com>;
> dev@dpdk.org
> Subject: [EXT] RE: [dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-
> secgw
> 
> External Email
> 
> ----------------------------------------------------------------------
> 
> 
> > This series introduces event-mode additions to ipsec-secgw. This
> > effort is based on the proposed changes for l2fwd-event and the
> > additions in l3fwd for event support.
> >
> > With this series, ipsec-secgw would be able to run in eventmode. The
> > worker thread (executing loop) would be receiving events and would be
> > submitting it back to the eventdev after the processing. This way,
> > multicore scaling and h/w assisted scheduling is achieved by making
> > use of the eventdev capabilities.
> >
> > Since the underlying event device would be having varying
> > capabilities, the worker thread could be drafted differently to maximize
> performance.
> > This series introduces usage of multiple worker threads, among which
> > the one to be used will be determined by the operating conditions and
> > the underlying device capabilities.
> >
> > For example, if an event device - eth device pair has Tx internal
> > port, then application can do tx_adapter_enqueue() instead of regular
> > event_enqueue(). So a thread making an assumption that the device pair
> > has internal port will not be the right solution for another pair. The
> > infrastructure added with these patches aims to help application to
> > have multiple worker threads, there by extracting maximum performance
> > from every device without affecting existing paths/use cases.
> >
> > The eventmode configuration is predefined. All packets reaching one
> > eth port will hit one event queue. All event queues will be mapped to
> > all event ports. So all cores will be able to receive traffic from all ports.
> > When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event
> > device will ensure the ordering. Ordering would be lost when tried in
> PARALLEL.
> >
> > Following command line options are introduced,
> >
> > --transfer-mode: to choose between poll mode & event mode
> > --schedule-type: to specify the scheduling type
> >                  (RTE_SCHED_TYPE_ORDERED/
> >                   RTE_SCHED_TYPE_ATOMIC/
> >                   RTE_SCHED_TYPE_PARALLEL)
> > --process-dir: outbound/inbound
> > --process-mode: app mode /driver mode
> >
> > The two s/w config options added to ipsec-secgw can be used in
> > benchmarking h/w performance,
> >
> 
> I didn't look at the actual code (yet), just cover letter, few quick questions
> below.
> 
> > 1. process-dir : states whether the direction is outbound/inbound.
> > This option aims to avoid an unnecessary check of determining whether
> > inbound/outbound processing need to be done on the packet. For each
> > option a different light weight worker thread would be executed.
> 
> Bur right now app can do both inbound and outbound simultaneously on the
> same core.
> I presume the default behavior will be preserved?

[Anoob] The existing behavior with poll mode thread is not touched. The current changes are only applicable when launched in event-mode.

> 
> >
> > 2. process-mode: states whether the application has to run in driver
> >                  mode or app mode.
> >
> > Driver-mode: This mode will have bare minimum changes in the application
> >              to support ipsec. There woudn't be any lookup etc done in
> >              the application. And for inline-protocol use case, the
> >              thread would resemble l2fwd as the ipsec processing would be
> >              done entirely in the h/w. This mode can be used to benchmark
> >              the raw performance of the h/w. All the application side
> >              steps (like lookup) can be redone based on the requirement
> >              of the end user. Hence the need for a mode which would
> >              report the raw performance.
> >
> > App-mode: This mode will have all the features currently implemented
> with
> >           ipsec-secgw (non librte_ipsec mode). All the lookups etc
> >           would follow the existing methods and would report numbers
> >           that can be compared against regular ipsec-secgw benchmark
> >           numbers.
> >
> > Example commands to execute ipsec-secgw in various modes on
> OCTEONTX2
> > platform,
> >
> > #Inbound driver mode
> > ./ipsec-secgw -w 0002:02:00.0,nb_ipsec_in_sa=128 -w
> > 0002:03:00.0,nb_ipsec_in_sa=128 -w 0002:04:00.0,nb_ipsec_in_sa=128 -w
> > 0002:07:00.0,nb_ipsec_in_sa=128 -w 0002:0e:00.0 -w 0002:10:00.1
> > --log-level=8 -c 0x7 – -P -p 0xf --config
> > "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f dpdk_internal/100g_4.3.cfg
> > --transfer-mode 1 --schedule-type 2 --process-mode 1 --process-dir 1
> 
> For all these parameters, I think better to use names (app/driver, etc.)
> instead of raw numbers.

[Anoob] Agreed. Will make this change when we submit the actual patches.

> 
> >
> > #Inbound app mode
> > ./ipsec-secgw -w 0002:02:00.0,nb_ipsec_in_sa=128 -w
> > 0002:03:00.0,nb_ipsec_in_sa=128 -w 0002:04:00.0,nb_ipsec_in_sa=128 -w
> > 0002:07:00.0,nb_ipsec_in_sa=128 -w 0002:0e:00.0 -w 0002:10:00.1
> > --log-level=8 -c 0x3f – -P -p 0xf --config
> > "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" - f dpdk_internal/100g_4.3.cfg
> > --transfer-mode 1 --schedule-type 2 --process-mode 0 --process-dir 1
> >
> > #Outbound driver mode
> > ./ipsec-secgw -w 0002:02:00.0 -w 0002:03:00.0 -w 0002:04:00.0 -w
> > 0002:07:00.0 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x1f – -
> > P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f
> > a-aes-gcm-new.cfg --transfer-mode 1 --schedule-type 2 --process-mode 1
> > --process-dir 0
> >
> > #Outbound app mode
> > ./ipsec-secgw -w 0002:02:00.0 -w 0002:03:00.0 -w 0002:04:00.0 -w
> > 0002:07:00.0 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x7f – -
> > P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f
> > a-aes-gcm-new.cfg --transfer-mode 1 --schedule-type 2 --process-mode 0
> > --process-dir 0
> >
> > This series is targeted for next release (20.02). This series doesn't
> > introduce any library change.
> 
> By 'library change' you mean that this new event-mode will be supported
> only by legacy code-path or ...?

[Anoob] All the changes are confined to 'examples/ipsec-secgw' directory. Right now, the worker threads make use of the existing routines in non-librte_ipsec mode. 

> 
> >And the decision to add eventmode additions in ipsec-secgw  was
> >approved by the Tech Board.
> >
> > Following are missing in the RFC. Will add it when sending patches.
> > 1. Documentation.
> > 2. More cleanup is needed. There are options that are added so that future
> >    expansion is not hindered. Need inputs from the community if there is
> use
> >    case for them.
> >
> > Following are planned features,
> > 1. Add burst mode workers.
> > 2. Add non tx internal port worker.
> > 3. Verify support for Rx core (the support is added but lack of h/w to
> verify).
> > 4. Add lookaside protocol support.
> >
> > Following are features that Marvell won't be attempting.
> > 1. Inline crypto support.
> > 2. Lookaside crypto support.
> 
> Ok so what mode is supported right now with this RFC?

[Anoob] Inline protocol support is added with the RFC.

> 
> >
> > For the features that Marvell won't be attempting, new workers can be
> > introduced by the respective stake holders.
> >
> > Anoob Joseph (13):
> >   examples/ipsec-secgw: add framework for eventmode helper
> >   examples/ipsec-secgw: add eventdev port-lcore link
> >   examples/ipsec-secgw: add Rx adapter support
> >   examples/ipsec-secgw: add Tx adapter support
> >   examples/ipsec-secgw: add routines to display config
> >   examples/ipsec-secgw: add routines to launch workers
> >   examples/ipsec-secgw: add support for internal ports
> >   examples/ipsec-secgw: add eventmode to ipsec-secgw
> >   examples/ipsec-secgw: add app inbound worker
> >   examples/ipsec-secgw: add app processing code
> >   examples/ipsec-secgw: add driver outbound worker
> >   examples/ipsec-secgw: add app outbound worker
> >   examples/ipsec-secgw: add cmd line option for bufs
> >
> >  examples/ipsec-secgw/Makefile       |    2 +
> >  examples/ipsec-secgw/event_helper.c | 1757
> > +++++++++++++++++++++++++++++++++++
> >  examples/ipsec-secgw/event_helper.h |  334 +++++++
> > examples/ipsec-secgw/ipsec-secgw.c  |  436 +++++++--
> >  examples/ipsec-secgw/ipsec-secgw.h  |   81 ++
> >  examples/ipsec-secgw/ipsec.c        |    4 +
> >  examples/ipsec-secgw/ipsec.h        |   30 +-
> >  examples/ipsec-secgw/ipsec_worker.c |  766 +++++++++++++++
> >  examples/ipsec-secgw/ipsec_worker.h |   39 +
> >  examples/ipsec-secgw/meson.build    |    4 +-
> >  examples/ipsec-secgw/sa.c           |   11 -
> >  11 files changed, 3360 insertions(+), 104 deletions(-)  create mode
> > 100644 examples/ipsec-secgw/event_helper.c
> >  create mode 100644 examples/ipsec-secgw/event_helper.h
> >  create mode 100644 examples/ipsec-secgw/ipsec-secgw.h
> >  create mode 100644 examples/ipsec-secgw/ipsec_worker.c
> >  create mode 100644 examples/ipsec-secgw/ipsec_worker.h
> >
> > --
> > 2.7.4