[PATCH v3 0/2] example/l3fwd: relax l3fwd rx RSS/Offload if needed

[PATCH v3 0/2] example/l3fwd: relax l3fwd rx RSS/Offload if needed

[Trevor Tao]

This series try to relax l3fwd rx RSS/Offload mode requirement if they
are not supported by underlying hw or virtual devices, there is an
option named relax_rx_mode added to enable this option.

Trevor Tao (2):
  examples/l3fwd: relax RSS requirement with option
  examples/l3fwd: relax the Offload requirement

 examples/l3fwd/l3fwd.h     | 12 ++++++++++--
 examples/l3fwd/l3fwd_em.h  |  2 +-
 examples/l3fwd/l3fwd_lpm.h |  2 +-
 examples/l3fwd/main.c      | 30 ++++++++++++++++++++++++++++++
 4 files changed, 42 insertions(+), 4 deletions(-)

-- 
2.34.1

[PATCH v3 1/2] examples/l3fwd: relax RSS requirement with option

[Trevor Tao]

Now the port Rx mq_mode had been set to RTE_ETH_MQ_RX_RSS
by default, but some hw and/or virtual interface does not
support the RSS and offload mode presupposed, e.g., some
virtio interfaces in the cloud don't support
RSS and the error msg may like:

virtio_dev_configure(): RSS support requested but not supported by
the device
Port0 dev_configure = -95

So to enable the l3fwd running in that environment, the Rx mode requirement
can be relaxed to reflect the hardware feature reality here, and the l3fwd
can run smoothly then.

An option named "relax-rx-mode" is added to enable the relax action
here, and it's disabled by default.

Signed-off-by: Trevor Tao <taozj888@163.com>
---
 examples/l3fwd/main.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/examples/l3fwd/main.c b/examples/l3fwd/main.c
index 6063eb1399..89ad546a5e 100644
--- a/examples/l3fwd/main.c
+++ b/examples/l3fwd/main.c
@@ -73,6 +73,7 @@ static enum L3FWD_LOOKUP_MODE lookup_mode;
 static int numa_on = 1; /**< NUMA is enabled by default. */
 static int parse_ptype; /**< Parse packet type using rx callback, and */
 			/**< disabled by default */
+static int relax_rx_mode; /**< Relax RX mode is disabled by default */
 static int per_port_pool; /**< Use separate buffer pools per port; disabled */
 			  /**< by default */
 
@@ -678,6 +679,7 @@ static const char short_options[] =
 #define CMD_LINE_OPT_MAX_PKT_LEN "max-pkt-len"
 #define CMD_LINE_OPT_HASH_ENTRY_NUM "hash-entry-num"
 #define CMD_LINE_OPT_PARSE_PTYPE "parse-ptype"
+#define CMD_LINE_OPT_RELAX_RX_MODE "relax-rx-mode"
 #define CMD_LINE_OPT_PER_PORT_POOL "per-port-pool"
 #define CMD_LINE_OPT_MODE "mode"
 #define CMD_LINE_OPT_EVENTQ_SYNC "eventq-sched"
@@ -705,6 +707,7 @@ enum {
 	CMD_LINE_OPT_MAX_PKT_LEN_NUM,
 	CMD_LINE_OPT_HASH_ENTRY_NUM_NUM,
 	CMD_LINE_OPT_PARSE_PTYPE_NUM,
+	CMD_LINE_OPT_RELAX_RX_MODE_NUM,
 	CMD_LINE_OPT_RULE_IPV4_NUM,
 	CMD_LINE_OPT_RULE_IPV6_NUM,
 	CMD_LINE_OPT_ALG_NUM,
@@ -728,6 +731,7 @@ static const struct option lgopts[] = {
 	{CMD_LINE_OPT_MAX_PKT_LEN, 1, 0, CMD_LINE_OPT_MAX_PKT_LEN_NUM},
 	{CMD_LINE_OPT_HASH_ENTRY_NUM, 1, 0, CMD_LINE_OPT_HASH_ENTRY_NUM_NUM},
 	{CMD_LINE_OPT_PARSE_PTYPE, 0, 0, CMD_LINE_OPT_PARSE_PTYPE_NUM},
+	{CMD_LINE_OPT_RELAX_RX_MODE, 0, 0, CMD_LINE_OPT_RELAX_RX_MODE_NUM},
 	{CMD_LINE_OPT_PER_PORT_POOL, 0, 0, CMD_LINE_OPT_PARSE_PER_PORT_POOL},
 	{CMD_LINE_OPT_MODE, 1, 0, CMD_LINE_OPT_MODE_NUM},
 	{CMD_LINE_OPT_EVENTQ_SYNC, 1, 0, CMD_LINE_OPT_EVENTQ_SYNC_NUM},
@@ -853,6 +857,11 @@ parse_args(int argc, char **argv)
 			parse_ptype = 1;
 			break;
 
+		case CMD_LINE_OPT_RELAX_RX_MODE_NUM:
+			printf("Relax rx mode is enabled\n");
+			relax_rx_mode = 1;
+			break;
+
 		case CMD_LINE_OPT_PARSE_PER_PORT_POOL:
 			printf("per port buffer pool is enabled\n");
 			per_port_pool = 1;
@@ -1260,6 +1269,13 @@ l3fwd_poll_resource_setup(void)
 		if (dev_info.max_rx_queues == 1)
 			local_port_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE;
 
+		/* relax the rx rss requirement */
+		if (relax_rx_mode && !local_port_conf.rx_adv_conf.rss_conf.rss_hf) {
+			printf("warning: modified the rx mq_mode to RTE_ETH_MQ_RX_NONE base on"
+					" device capability\n");
+				local_port_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE;
+		}
+
 		if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
 				port_conf.rx_adv_conf.rss_conf.rss_hf) {
 			printf("Port %u modified RSS hash function based on hardware support,"
-- 
2.34.1

Re: [PATCH v3 1/2] examples/l3fwd: relax RSS requirement with option

[Stephen Hemminger]

On Sun, 12 Nov 2023 07:16:10 +0000
Trevor Tao <taozj888@163.com> wrote:

> Now the port Rx mq_mode had been set to RTE_ETH_MQ_RX_RSS
> by default, but some hw and/or virtual interface does not
> support the RSS and offload mode presupposed, e.g., some
> virtio interfaces in the cloud don't support
> RSS and the error msg may like:
> 
> virtio_dev_configure(): RSS support requested but not supported by
> the device
> Port0 dev_configure = -95
> 
> So to enable the l3fwd running in that environment, the Rx mode requirement
> can be relaxed to reflect the hardware feature reality here, and the l3fwd
> can run smoothly then.
> 
> An option named "relax-rx-mode" is added to enable the relax action
> here, and it's disabled by default.
> 
> Signed-off-by: Trevor Tao <taozj888@163.com>

Doesn't need to be an option. The application can just look at the offload
flags and quickly determine that if multiple queues are being used than
RSS is required. If device doesn't support RSS, then fallback to single
queue with a warning.

Re:Re: [PATCH v3 1/2] examples/l3fwd: relax RSS requirement with option

[taozj888]

[-- Attachment #1: Type: text/plain, Size: 1896 bytes --]

At 2023-11-13 07:33:49, "Stephen Hemminger" <stephen@networkplumber.org> wrote:
>On Sun, 12 Nov 2023 07:16:10 +0000
>Trevor Tao <taozj888@163.com> wrote:
>
>> Now the port Rx mq_mode had been set to RTE_ETH_MQ_RX_RSS
>> by default, but some hw and/or virtual interface does not
>> support the RSS and offload mode presupposed, e.g., some
>> virtio interfaces in the cloud don't support
>> RSS and the error msg may like:
>> 
>> virtio_dev_configure(): RSS support requested but not supported by
>> the device
>> Port0 dev_configure = -95
>> 
>> So to enable the l3fwd running in that environment, the Rx mode requirement
>> can be relaxed to reflect the hardware feature reality here, and the l3fwd
>> can run smoothly then.
>> 
>> An option named "relax-rx-mode" is added to enable the relax action
>> here, and it's disabled by default.
>> 
>> Signed-off-by: Trevor Tao <taozj888@163.com>
>
>Doesn't need to be an option. The application can just look at the offload
>flags and quickly determine that if multiple queues are being used than
>RSS is required. If device doesn't support RSS, then fallback to single

>queue with a warning.


Hi Stephen:


I think the option here was suggested/required by Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>.
The discussion thread is listed below:


">>>>Should we probably instead have a new commnad-line option to explicitly 
>>>>disable RSS?
>>> 
>>>>Something like: '--no-rss' or so?
>>> Trevor: the RSS capability for a certain port was got by the rte_eth_dev_info_get() automatically, and we think the user should not care about its status beforehand, but if it's missing, a warning notification for the degrade here would be proposed to make it run smoothly.
>>
>>Personally, I still think it would be better the user will
>>have an ability to disable it explicitly.
>>Same as l3fwd does now with 'parse-ptype'.

>>
"


Thanks,


Trevor Tao

[-- Attachment #2: Type: text/html, Size: 3157 bytes --]

RE: Re: [PATCH v3 1/2] examples/l3fwd: relax RSS requirement with option

[Morten Brørup]

[-- Attachment #1: Type: text/plain, Size: 2468 bytes --]

From: taozj888 [mailto:taozj888@163.com] 
Sent: Monday, 13 November 2023 12.47



At 2023-11-13 07:33:49, "Stephen Hemminger" <stephen@networkplumber.org> wrote:

>On Sun, 12 Nov 2023 07:16:10 +0000
>Trevor Tao <taozj888@163.com> wrote:
> 
>> Now the port Rx mq_mode had been set to RTE_ETH_MQ_RX_RSS
>> by default, but some hw and/or virtual interface does not
>> support the RSS and offload mode presupposed, e.g., some
>> virtio interfaces in the cloud don't support
>> RSS and the error msg may like:
>> 
>> virtio_dev_configure(): RSS support requested but not supported by
>> the device
>> Port0 dev_configure = -95
>> 
>> So to enable the l3fwd running in that environment, the Rx mode requirement
>> can be relaxed to reflect the hardware feature reality here, and the l3fwd
>> can run smoothly then.
>> 
>> An option named "relax-rx-mode" is added to enable the relax action
>> here, and it's disabled by default.
>> 
>> Signed-off-by: Trevor Tao <taozj888@163.com>
> 
>Doesn't need to be an option. The application can just look at the offload
>flags and quickly determine that if multiple queues are being used than
>RSS is required. If device doesn't support RSS, then fallback to single
>queue with a warning.
 
Hi Stephen:
 
I think the option here was suggested/required by Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>.
The discussion thread is listed below:
 
">>>>Should we probably instead have a new commnad-line option to explicitly 
>>>>disable RSS?
>>> 
>>>>Something like: '--no-rss' or so?
>>> Trevor: the RSS capability for a certain port was got by the rte_eth_dev_info_get() automatically, and we think the user should not care about its status beforehand, but if it's missing, a warning notification for the degrade here would be proposed to make it run smoothly.
>> 
>>Personally, I still think it would be better the user will
>>have an ability to disable it explicitly.
>>Same as l3fwd does now with 'parse-ptype'.
>> 
"
 
[Morten]: The "relax-rx-mode" does not explicitly disable RSS, it enables the behavior Stephen suggested: If RSS is not supported by the device, fallback to single queue with a warning. We don't need an option for this; it should be the default behavior.
[Morten]: I think Konstantin was asking for an option "no-rss" (or "disable-rss") to force use of single queue (disable RSS), regardless if the device supports RSS or not.
 
Thanks,
 
Trevor Tao

[-- Attachment #2: Type: text/html, Size: 8591 bytes --]

RE: Re: [PATCH v3 1/2] examples/l3fwd: relax RSS requirement with option

[Konstantin Ananyev]

>> Now the port Rx mq_mode had been set to RTE_ETH_MQ_RX_RSS
>> by default, but some hw and/or virtual interface does not
>> support the RSS and offload mode presupposed, e.g., some
>> virtio interfaces in the cloud don't support
>> RSS and the error msg may like:
>> 
>> virtio_dev_configure(): RSS support requested but not supported by
>> the device
>> Port0 dev_configure = -95
>> 
>> So to enable the l3fwd running in that environment, the Rx mode requirement
>> can be relaxed to reflect the hardware feature reality here, and the l3fwd
>> can run smoothly then.
>> 
>> An option named "relax-rx-mode" is added to enable the relax action
>> here, and it's disabled by default.
>> 
>> Signed-off-by: Trevor Tao <mailto:taozj888@163.com>
>
>Doesn't need to be an option. The application can just look at the offload
>flags and quickly determine that if multiple queues are being used than
>RSS is required. If device doesn't support RSS, then fallback to single
>queue with a warning.
>
>Hi Stephen:
>
>I think the option here was suggested/required by Konstantin Ananyev <mailto:konstantin.v.ananyev@yandex.ru>.
>The discussion thread is listed below:
>
">>>>Should we probably instead have a new commnad-line option to explicitly 
>>>>disable RSS?
>>> 
>>>>Something like: '--no-rss' or so?
>>> Trevor: the RSS capability for a certain port was got by the rte_eth_dev_info_get() automatically, and we think the user should not care about its status beforehand, but if it's missing, a warning notification for the degrade here would be proposed to make it run smoothly.
>>
>>Personally, I still think it would be better the user will
>>have an ability to disable it explicitly.
>>Same as l3fwd does now with 'parse-ptype'.
>>
>
>[Morten]: The "relax-rx-mode" does not explicitly disable RSS, it enables the behavior Stephen suggested: If RSS is not supported by the device, fallback to single queue with a warning. We don't need an option >for this; it should be the default behavior.
>[Morten]: I think Konstantin was asking for an option "no-rss" (or "disable-rss") to force use of single queue (disable RSS), regardless if the device supports RSS or not.

Yes, that what was my initial suggestion: let people explicitly disable RSS if they want to.
BTW,  I think it wouldn't change number of HW queues in any way.
It would just disable distribution of packets among HW queues based on RSS.
Konstantin 

Re:RE: Re: [PATCH v3 1/2] examples/l3fwd: relax RSS requirement with option

[taozj888]

[-- Attachment #1: Type: text/plain, Size: 2874 bytes --]

Hi all:

At 2023-11-13 21:19:14, "Konstantin Ananyev" <konstantin.ananyev@huawei.com> wrote:
>
>
>>> Now the port Rx mq_mode had been set to RTE_ETH_MQ_RX_RSS
>>> by default, but some hw and/or virtual interface does not
>>> support the RSS and offload mode presupposed, e.g., some
>>> virtio interfaces in the cloud don't support
>>> RSS and the error msg may like:
>>> 
>>> virtio_dev_configure(): RSS support requested but not supported by
>>> the device
>>> Port0 dev_configure = -95
>>> 
>>> So to enable the l3fwd running in that environment, the Rx mode requirement
>>> can be relaxed to reflect the hardware feature reality here, and the l3fwd
>>> can run smoothly then.
>>> 
>>> An option named "relax-rx-mode" is added to enable the relax action
>>> here, and it's disabled by default.
>>> 
>>> Signed-off-by: Trevor Tao <mailto:taozj888@163.com>
>>
>>Doesn't need to be an option. The application can just look at the offload
>>flags and quickly determine that if multiple queues are being used than
>>RSS is required. If device doesn't support RSS, then fallback to single
>>queue with a warning.
>>
>>Hi Stephen:
>>
>>I think the option here was suggested/required by Konstantin Ananyev <mailto:konstantin.v.ananyev@yandex.ru>.
>>The discussion thread is listed below:
>>
>">>>>Should we probably instead have a new commnad-line option to explicitly 
>>>>>disable RSS?
>>>> 
>>>>>Something like: '--no-rss' or so?
>>>> Trevor: the RSS capability for a certain port was got by the rte_eth_dev_info_get() automatically, and we think the user should not care about its status beforehand, but if it's missing, a warning notification for the degrade here would be proposed to make it run smoothly.
>>>
>>>Personally, I still think it would be better the user will
>>>have an ability to disable it explicitly.
>>>Same as l3fwd does now with 'parse-ptype'.
>>>
>>
>>[Morten]: The "relax-rx-mode" does not explicitly disable RSS, it enables the behavior Stephen suggested: If RSS is not supported by the device, fallback to single queue with a warning. We don't need an option >for this; it should be the default behavior.
>>[Morten]: I think Konstantin was asking for an option "no-rss" (or "disable-rss") to force use of single queue (disable RSS), regardless if the device supports RSS or not.
>
>Yes, that what was my initial suggestion: let people explicitly disable RSS if they want to.
>BTW,  I think it wouldn't change number of HW queues in any way.
>It would just disable distribution of packets among HW queues based on RSS.
>Konstantin 

>


Thanks for your comments. I addressed your ideas with a v4 patch series.
1. A new option named "disable-rss" was added to enable the "diable-rss" case;
2. A new option  named "relax-rx-offload" was added to enable the relax rx offload case; and the SW IPv4 hdr cksum was added if needed;


Trevor Tao 


Trevor Tao 

[-- Attachment #2: Type: text/html, Size: 3500 bytes --]

[PATCH v3 2/2] examples/l3fwd: relax the Offload requirement

[Trevor Tao]

Now the port Rx offload mode is set to RTE_ETH_RX_OFFLOAD_CHECKSUM
by default, but some hw and/or virtual interface does not support
the offload mode presupposed, e.g., some virtio interfaces in
the cloud may only partly support RTE_ETH_RX_OFFLOAD_UDP_CKSUM/
RTE_ETH_RX_OFFLOAD_TCP_CKSUM,
but not RTE_ETH_RX_OFFLOAD_IPV4_CKSUM, and the error msg here:

Ethdev port_id=0 requested Rx offloads 0xe does not match Rx offloads
capabilities 0x201d in rte_eth_dev_configure()

So to enable the l3fwd running in that environment, the Rx mode requirement
can be relaxed to reflect the hardware feature reality here, and the l3fwd
can run smoothly then.
A warning msg would be provided to user in case it happens here.

On the other side, enabling the software cksum check in case missing the
hw support.

The relax action for rx cksum offload is just enabled when relax_rx_mode is
true which is false by default.

Signed-off-by: Trevor Tao <taozj888@163.com>
---
 examples/l3fwd/l3fwd.h     | 12 ++++++++++--
 examples/l3fwd/l3fwd_em.h  |  2 +-
 examples/l3fwd/l3fwd_lpm.h |  2 +-
 examples/l3fwd/main.c      | 14 ++++++++++++++
 4 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/examples/l3fwd/l3fwd.h b/examples/l3fwd/l3fwd.h
index b55855c932..e7ae0e5834 100644
--- a/examples/l3fwd/l3fwd.h
+++ b/examples/l3fwd/l3fwd.h
@@ -159,7 +159,7 @@ send_single_packet(struct lcore_conf *qconf,
 
 #ifdef DO_RFC_1812_CHECKS
 static inline int
-is_valid_ipv4_pkt(struct rte_ipv4_hdr *pkt, uint32_t link_len)
+is_valid_ipv4_pkt(struct rte_ipv4_hdr *pkt, uint32_t link_len, uint64_t ol_flags)
 {
 	/* From http://www.rfc-editor.org/rfc/rfc1812.txt section 5.2.2 */
 	/*
@@ -170,7 +170,15 @@ is_valid_ipv4_pkt(struct rte_ipv4_hdr *pkt, uint32_t link_len)
 		return -1;
 
 	/* 2. The IP checksum must be correct. */
-	/* this is checked in H/W */
+	/* if this is not checked in H/W, check it. */
+	if ((ol_flags & RTE_MBUF_F_RX_IP_CKSUM_MASK) == RTE_MBUF_F_RX_IP_CKSUM_NONE) {
+		uint16_t actual_cksum, expected_cksum;
+		actual_cksum = pkt->hdr_checksum;
+		pkt->hdr_checksum = 0;
+		expected_cksum = rte_ipv4_cksum(pkt);
+		if (actual_cksum != expected_cksum)
+			return -2;
+	}
 
 	/*
 	 * 3. The IP version number must be 4. If the version number is not 4
diff --git a/examples/l3fwd/l3fwd_em.h b/examples/l3fwd/l3fwd_em.h
index 7d051fc076..1fee2e2e6c 100644
--- a/examples/l3fwd/l3fwd_em.h
+++ b/examples/l3fwd/l3fwd_em.h
@@ -20,7 +20,7 @@ l3fwd_em_handle_ipv4(struct rte_mbuf *m, uint16_t portid,
 
 #ifdef DO_RFC_1812_CHECKS
 	/* Check to make sure the packet is valid (RFC1812) */
-	if (is_valid_ipv4_pkt(ipv4_hdr, m->pkt_len) < 0) {
+	if (is_valid_ipv4_pkt(ipv4_hdr, m->pkt_len, m->ol_flags) < 0) {
 		rte_pktmbuf_free(m);
 		return BAD_PORT;
 	}
diff --git a/examples/l3fwd/l3fwd_lpm.h b/examples/l3fwd/l3fwd_lpm.h
index c61b969584..4ee61e8d88 100644
--- a/examples/l3fwd/l3fwd_lpm.h
+++ b/examples/l3fwd/l3fwd_lpm.h
@@ -22,7 +22,7 @@ l3fwd_lpm_simple_forward(struct rte_mbuf *m, uint16_t portid,
 
 #ifdef DO_RFC_1812_CHECKS
 		/* Check to make sure the packet is valid (RFC1812) */
-		if (is_valid_ipv4_pkt(ipv4_hdr, m->pkt_len) < 0) {
+		if (is_valid_ipv4_pkt(ipv4_hdr, m->pkt_len, m->ol_flags) < 0) {
 			rte_pktmbuf_free(m);
 			return;
 		}
diff --git a/examples/l3fwd/main.c b/examples/l3fwd/main.c
index 89ad546a5e..2b815375a9 100644
--- a/examples/l3fwd/main.c
+++ b/examples/l3fwd/main.c
@@ -1285,6 +1285,20 @@ l3fwd_poll_resource_setup(void)
 				local_port_conf.rx_adv_conf.rss_conf.rss_hf);
 		}
 
+		/* relax the rx offload requirement */
+		if ((local_port_conf.rxmode.offloads & dev_info.rx_offload_capa) !=
+			local_port_conf.rxmode.offloads) {
+			printf("Port %u requested Rx offloads 0x%"PRIx64" does not"
+				" match Rx offloads capabilities 0x%"PRIx64"\n",
+				portid, local_port_conf.rxmode.offloads,
+				dev_info.rx_offload_capa);
+			if (relax_rx_mode) {
+				local_port_conf.rxmode.offloads &= dev_info.rx_offload_capa;
+				printf("warning: modified the rx offload to 0x%"PRIx64" based on device"
+				" capability\n", local_port_conf.rxmode.offloads);
+			}
+		}
+
 		ret = rte_eth_dev_configure(portid, nb_rx_queue,
 					(uint16_t)n_tx_queue, &local_port_conf);
 		if (ret < 0)
-- 
2.34.1