Re: [dpdk-dev] [EXT] RE: [PATCH v4 00/15] add eventmode to ipsec-secgw

[Lukas Bartosik <lbartosik@marvell.com>]

Hi Akhil,

Please see my answer below.

Thanks,
Lukasz

On 24.02.2020 14:40, Akhil Goyal wrote:
> External Email
> 
> ----------------------------------------------------------------------
> Hi Anoob/Lukasz,
> 
>>
>> This series introduces event-mode additions to ipsec-secgw.
>>
>> With this series, ipsec-secgw would be able to run in eventmode. The
>> worker thread (executing loop) would be receiving events and would be
>> submitting it back to the eventdev after the processing. This way,
>> multicore scaling and h/w assisted scheduling is achieved by making use
>> of the eventdev capabilities.
>>
>> Since the underlying event device would be having varying capabilities,
>> the worker thread could be drafted differently to maximize performance.
>> This series introduces usage of multiple worker threads, among which the
>> one to be used will be determined by the operating conditions and the
>> underlying device capabilities.
>>
>> For example, if an event device - eth device pair has Tx internal port,
>> then application can do tx_adapter_enqueue() instead of regular
>> event_enqueue(). So a thread making an assumption that the device pair
>> has internal port will not be the right solution for another pair. The
>> infrastructure added with these patches aims to help application to have
>> multiple worker threads, there by extracting maximum performance from
>> every device without affecting existing paths/use cases.
>>
>> The eventmode configuration is predefined. All packets reaching one eth
>> port will hit one event queue. All event queues will be mapped to all
>> event ports. So all cores will be able to receive traffic from all ports.
>> When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event
>> device
>> will ensure the ordering. Ordering would be lost when tried in PARALLEL.
>>
>> Following command line options are introduced,
>>
>> --transfer-mode: to choose between poll mode & event mode
>> --event-schedule-type: to specify the scheduling type
>>                  (RTE_SCHED_TYPE_ORDERED/
>>                   RTE_SCHED_TYPE_ATOMIC/
>>                   RTE_SCHED_TYPE_PARALLEL)
>>
>> Additionally the event mode introduces two modes of processing packets:
>>
>> Driver-mode: This mode will have bare minimum changes in the application
>>              to support ipsec. There woudn't be any lookup etc done in
>>              the application. And for inline-protocol use case, the
>>              thread would resemble l2fwd as the ipsec processing would be
>>              done entirely in the h/w. This mode can be used to benchmark
>>              the raw performance of the h/w. All the application side
>>              steps (like lookup) can be redone based on the requirement
>>              of the end user. Hence the need for a mode which would
>>              report the raw performance.
>>
>> App-mode: This mode will have all the features currently implemented with
>>           ipsec-secgw (non librte_ipsec mode). All the lookups etc
>>           would follow the existing methods and would report numbers
>>           that can be compared against regular ipsec-secgw benchmark
>>           numbers.
>>
>> The driver mode is selected with existing --single-sa option
>> (used also by poll mode). When --single-sa option is used
>> in conjution with event mode then index passed to --single-sa
>> is ignored.
>>
>> Example commands to execute ipsec-secgw in various modes on OCTEON TX2
>> platform,
>>
>> #Inbound and outbound app mode
>> ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w
>> 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-
>> level=8 -c 0x1 -- -P -p 0x3 -u 0x1 --config "(1,0,0),(0,0,0)" -f aes-gcm.cfg --
>> transfer-mode event --event-schedule-type parallel
>>
> 
> What is the need of adding the port queue core mapping in case of event.
> In case of event, all queues are given to eventdev and there is no need for specifying such specific mapping. In l3fwd also this was not done.
> 

[Lukasz] You are right port,queue,core mapping is not needed in case of event mode. I will remove --config option from being used in event mode.
Number of ports to be initialized will be derived from the port mask and nb_rx_queues per port will be set to 1.

>> #Inbound and outbound driver mode
>> ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w
>> 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-
>> level=8 -c 0x1 -- -P -p 0x3 -u 0x1 --config "(1,0,0),(0,0,0)" -f aes-gcm.cfg --
>> transfer-mode event --event-schedule-type parallel --single-sa 0
>>
>> This series adds non burst tx internal port workers only. It provides infrastructure
>> for non internal port workers, however does not define any. Also, only inline
>> ipsec
>> protocol mode is supported by the worker threads added.
>>
>> Following are planned features,
>> 1. Add burst mode workers.
>> 2. Add non internal port workers.
>> 3. Verify support for Rx core (the support is added but lack of h/w to verify).
>> 4. Add lookaside protocol support.
>>
>> Following are features that Marvell won't be attempting.
>> 1. Inline crypto support.
>> 2. Lookaside crypto support.
>>
>> For the features that Marvell won't be attempting, new workers can be
>> introduced by the respective stake holders.
>>
>> This series is tested on Marvell OCTEON TX2.
>> This series is targeted for 20.05 release.
>>
>> Changes in v4:
>> * Update ipsec-secgw documentation to describe the new options as well as
>>   event mode support.
>> * In event mode reserve number of crypto queues equal to number of eth ports
>>   in order to meet inline protocol offload requirements.
>> * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool
>>   and include fragments table size into the calculation.
>> * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static keyword
>>   from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c.
>> * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), check_sp()
>>   and prepare_out_sessions_tbl() functions as a result of changes introduced
>>   by SAD feature.
>> * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx
>>   is created with rte_zmalloc.
>> * Minor cleanup enhancements:
>> - In eh_set_default_conf_eventdev() function in event_helper.c put definition
>>   of int local vars in one line, remove invalid comment, put
>>   "eventdev_config->ev_queue_mode = RTE_EVENT_QUEUE_CFG_ALL_TYPES"
>> in one line
>>   instead of two.
>> - Remove extern "C" from event_helper.h.
>> - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() and
>>   eh_dev_has_tx_internal_port() functions in event_helper.c.
>> - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c.
>> - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec-
>> secgw.h,
>>   remove #include <rte_hash.h>.
>> - Remove not needed includes in ipsec_worker.c.
>> - Remove expired todo from ipsec_worker.h.
>>
>> Changes in v3:
>> * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c
>>   including minor rework.
>> * Rename --schedule-type option to --event-schedule-type.
>> * Replace macro UNPROTECTED_PORT with static inline function
>>   is_unprotected_port().
>> * Move definitions of global variables used by multiple modules
>>   to .c files and add externs in .h headers.
>> * Add eh_check_conf() which validates ipsec-secgw configuration
>>   for event mode.
>> * Add dynamic calculation of number of buffers in a pool based
>>   on number of cores, ports and crypto queues.
>> * Fix segmentation fault in event mode driver worker which happens
>>   when there are no inline outbound sessions configured.
>> * Remove change related to updating number of crypto queues
>>   in cryptodevs_init(). The update of crypto queues will be handled
>>   in a separate patch.
>> * Fix compilation error on 32-bit platforms by using userdata instead
>>   of udata64 from rte_mbuf.
>>
>> Changes in v2:
>> * Remove --process-dir option. Instead use existing unprotected port mask
>>   option (-u) to decide wheter port handles inbound or outbound traffic.
>> * Remove --process-mode option. Instead use existing --single-sa option
>>   to select between app and driver modes.
>> * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread.
>> * Fix passing of req_rx_offload flags to create_default_ipsec_flow().
>> * Move destruction of flows to a location where eth ports are stopped
>>   and closed.
>> * Print error and exit when event mode --schedule-type option is used
>>   in poll mode.
>> * Reduce number of goto statements replacing them with loop constructs.
>> * Remove sec_session_fixed table and replace it with locally build
>>   table in driver worker thread. Table is indexed by port identifier
>>   and holds first inline session pointer found for a given port.
>> * Print error and exit when sessions other than inline are configured
>>   in event mode.
>> * When number of event queues is less than number of eth ports then
>>   map all eth ports to one event queue.
>> * Cleanup and minor improvements in code as suggested by Konstantin
>>
>> Ankur Dwivedi (1):
>>   examples/ipsec-secgw: add default rte flow for inline Rx
>>
>> Anoob Joseph (5):
>>   examples/ipsec-secgw: add framework for eventmode helper
>>   examples/ipsec-secgw: add eventdev port-lcore link
>>   examples/ipsec-secgw: add Rx adapter support
>>   examples/ipsec-secgw: add Tx adapter support
>>   examples/ipsec-secgw: add routines to display config
>>
>> Lukasz Bartosik (9):
>>   examples/ipsec-secgw: add routines to launch workers
>>   examples/ipsec-secgw: add support for internal ports
>>   examples/ipsec-secgw: add event helper config init/uninit
>>   examples/ipsec-secgw: add eventmode to ipsec-secgw
>>   examples/ipsec-secgw: add driver mode worker
>>   examples/ipsec-secgw: add app mode worker
>>   examples/ipsec-secgw: make number of buffers dynamic
>>   doc: add event mode support to ipsec-secgw
>>   examples/ipsec-secgw: reserve crypto queues in event mode
>>
>>  doc/guides/sample_app_ug/ipsec_secgw.rst |  138 ++-
>>  examples/ipsec-secgw/Makefile            |    2 +
>>  examples/ipsec-secgw/event_helper.c      | 1812
>> ++++++++++++++++++++++++++++++
>>  examples/ipsec-secgw/event_helper.h      |  327 ++++++
>>  examples/ipsec-secgw/ipsec-secgw.c       |  463 ++++++--
>>  examples/ipsec-secgw/ipsec-secgw.h       |   88 ++
>>  examples/ipsec-secgw/ipsec.c             |    5 +-
>>  examples/ipsec-secgw/ipsec.h             |   53 +-
>>  examples/ipsec-secgw/ipsec_worker.c      |  638 +++++++++++
>>  examples/ipsec-secgw/ipsec_worker.h      |   35 +
>>  examples/ipsec-secgw/meson.build         |    6 +-
>>  examples/ipsec-secgw/sa.c                |   21 +-
>>  examples/ipsec-secgw/sad.h               |    5 -
>>  13 files changed, 3464 insertions(+), 129 deletions(-)
>>  create mode 100644 examples/ipsec-secgw/event_helper.c
>>  create mode 100644 examples/ipsec-secgw/event_helper.h
>>  create mode 100644 examples/ipsec-secgw/ipsec-secgw.h
>>  create mode 100644 examples/ipsec-secgw/ipsec_worker.c
>>  create mode 100644 examples/ipsec-secgw/ipsec_worker.h
>>
>> --
>> 2.7.4
>