Re: [dpdk-dev] [RFC PATCH v1] regexdev: introduce regexdev subsystem

[Jerin Jacob Kollanukkaran <jerinj@marvell.com>]

Hi Xiang,

Sorry for delay in response(Was busy with 19.11 proposal deadline). Please see inline.
 
> 
> Reply to Xiang's queries in main thread:
> 
> Hi all,
> 
> Some questions regarding APIs. Could you please give more insights?
> 
> 1) rte_regex_ops
>       a) rsp_flags
>       These two flags RTE_REGEX_OPS_RSP_PMI_SOJ_F and
> RTE_REGEX_OPS_RSP_PMI_EOJ_F are used for cross buffer scan.
>       RTE_REGEX_OPS_RSP_PMI_EOJ_F tells whether we have a partial match
> at the end of current buffer after scan.
>       What's the purpose of having RTE_REGEX_OPS_RSP_PMI_SOJ_F?
> 
> [Jerin] Since we need three states to represent partial match buffer,
> RTE_REGEX_OPS_RSP_PMI_SOJ_F to
> represent start of the buffer, intermediate buffers with no flag, and end of
> the buffer with RTE_REGEX_OPS_RSP_PMI_EOJ

> [Xiang] How could a user leverage these flags for matching? Suppose a large
> buffer is divided into multiple chunks. Will RTE_REGEX_OPS_RSP_PMI_SOJ_F
> cause an early quit once it isn't set after scan the first chunk. Similarly,
> RTE_REGEX_OPS_RSP_PMI_EOJ tells a user whether to stop matching future
> buffers after finish the last chunk?

Let me describe with an example,

Assume,
1) struct rte_regex_dev_info:: max_payload_size set to 1024
2) rte_regex_dev_config:: dev_cfg_flags configured with RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F
3) Device programmed with matching "hello\s+world" pattern
4) user enqueue struct rte_regex_ops:: buf_addr point following "data" and struct rte_regex_op:: scan_size = 1024

data[0..1021] = data don’t have hello world pattern
data[1022] = 'h'
data[1023] = 'e'

5) user enqueue struct rte_regex_ops:: buf_addr point following "data" and struct rte_regex_op:: scan_size = 9

data[0] = 'l'
data[1] = 'l'
data[2] = 'o'
data[3] = ' '
data[4] = 'w'
data[5] = 'o'
data[6] = 'r'
data[7] = 'l'
data[8] = 'd'

If so,

Response to 4) will be RTE_REGEX_OPS_RSP_PMI_SOJ_F in rte_regex_ops:: rsp_flags on dequeue
Where rte_regex_match:: offset is 1022 and len 2

Response to 5) will be RTE_REGEX_OPS_RSP_PMI_EOJ_F in rte_regex_ops:: rsp_flags on dequeue
Where rte_regex_match:: offset is 0 and len 9


> 
>       RTE_REGEX_OPS_RSP_MAX_PREFIX_F: This looks like a definition for a
> specific hardware implementation. I am wondering what this PREFIX refers
> to:)?
> 
> [Jerin] Yes. Looks like it is for hardware specific implementation. Introduced
> rte_regex_dev_attr_set/get functions to make it portable and
> To add new implementation specific fields.
> For example, if a rule is
> /ABCDEF.*XYZ/, ABCD is considered the prefix, and EF.*XYZ is considered the
> factor. The prefix is a literal
> string, while the factor can contain complex regular expression constructs. As
> a result, rule matching occurs in
> two stages: prefix matching and factor matching.
> 
>       b)  user_id or user_ptr
>       Under what kind of circumstances should an application pass value into
> these variables for enqueue and dequeuer operations?
> 
> [Jerin] Just like rte_crypto_ops, struct rte_regex_ops also allocated using
> mempool normally, on enqueue, user can specify user_id
> If needed to in order identify the op on dequeue if required. The use case
> could be to store the sequence number from application
> POV or storing the mbuf ptr in which pattern is requested etc.
> 
> 
>  2) rte_regex_match
>       a) offset; /**< Starting Byte Position for matched rule. */ and  uint16_t
> len; /**< Length of match in bytes */
>       Looks like the matching offset is defined as *starting matching offset*
> instead of *end matching offset*, e.g. report the offset of "a" instead of "c"
> for pattern "abc".
>       If so, this makes it hard to integrate software regex libraries such as
> Hyperscan and RE2 as they only report *end matching offset* without length
> of match.
>       Although Hyperscan has API for *starting matching offset*, it only delivers
> partial syntax support. So I think we have to define *end of matching offset*
> for software solutions.
> 
> [Jerin] I understand the hyperscan's HS_FLAG_SOM_LEFTMOST tradeoffs. I
> thought application would need always the length of the match.
> Probably we will see how other HW implementation (from Mellanox) etc. We
> will try to abstract it, probably we can make it as function of "user
> requested".
> [Xiang] Yes, it will be good to make it per user request. At least from
> Hyperscan user's point of view, start of match and match length are not
> mandatory.

OK. I think, we can introduce RTE_REGEX_DEV_CFG_MATCH_AS_START
In device configure.

Since offset+len == end, we can introduce following generic inline function.

static inline 
rte_regex_match_end(truct rte_regex_match *match)
{
	match->offset + match->len;
}

Example:  pattern to match is  "hello\s+world"  and data is following
data[4] = 'h'
data[5] = 'e'
data[6] = 'l'
data[7] = 'l'
data[8] = 'o'
data[9] = ' '
data[10] = 'w'
data[11] = 'o'
data[12] = 'r'
data[13] = 'l'
data[14] = 'd'

if device is configured with RTE_REGEX_DEV_CFG_MATCH_AS_START
match->offset returns 4
match->len returns 11

if device is NOT configured with RTE_REGEX_DEV_CFG_MATCH_AS_START
driver MAY return the following(in hyperscan case)
match->offset returns 0
match->len returns 11 + 4

In both case(irrespective of flags, to make application life easy) rte_regex_match_end() would return 15.
If application demands for MATCH_AS_START then driver can return match->offset returns 4 and match->len returns 11
Aka set HS_FLAG_SOM_LEFTMOST in hyperscan driver, But application should use rte_regex_match_end()
for finding the end of the match. To make, work in all cases.

Is it OK? 

> 
> 3)  rte_regex_rule_db_update()
>     Does this mean we can dynamically add or delete rules for an already
> generated database without recompile from scratch for hardware Regex
> implementation?
>     If so, this isn't possible for software solutions as they don't support
> dynamic database update and require recompile.
> 
> [Jerin] rte_regex_rule_db_update() internally it would call recompile
> function for both HW and SW.
> See rte_regex_dev_config::rule_db in rte_regex_dev_configure() for
> precompiled rule database case.
> [Xiang] OK, sounds like we have to save the original rule-set for the device in
> order to do recompile. I see both ADD and REMOVE operators from
> rte_regex_rule.
> For rules with REMOVE operator, what's the expected behavior to handle
> them for the old rule-set? Do we need to go through the old rule-set and
> remove corresponding rules before doing recompile?

Yes.