From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 250BD42CB1; Wed, 14 Jun 2023 04:22:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 02C2240E0F; Wed, 14 Jun 2023 04:22:50 +0200 (CEST) Received: from mail.loongson.cn (mail.loongson.cn [114.242.206.163]) by mails.dpdk.org (Postfix) with ESMTP id D23BE40DDB for ; Wed, 14 Jun 2023 04:22:48 +0200 (CEST) Received: from loongson.cn (unknown [10.20.42.90]) by gateway (Coremail) with SMTP id _____8BxLut2JIlk0+kEAA--.10708S3; Wed, 14 Jun 2023 10:22:46 +0800 (CST) Received: from [10.20.42.90] (unknown [10.20.42.90]) by localhost.localdomain (Coremail) with SMTP id AQAAf8DxVeR1JIlkCwgaAA--.9178S3; Wed, 14 Jun 2023 10:22:45 +0800 (CST) Subject: Re: [RFC] app/testpmd: use RSS conf from software when configuring DCB To: Ferruh Yigit , aman.deep.singh@intel.com, yuying.zhang@intel.com Cc: dev@dpdk.org, maobibo@loongson.cn, Qiming Yang , Wenjun Wu , "Zhang, Qi Z" References: <20230412095239.1709477-1-zhoumin@loongson.cn> <70e19912-ab95-ea91-b77f-028d2bbf4ef3@loongson.cn> <84cee6de-3b42-11ac-202e-22ff24ee289e@loongson.cn> <38f4bfd4-fced-bd0b-6261-1b34cd894dda@amd.com> From: zhoumin Message-ID: <7c4a1290-f47b-2648-4c67-085f8e58ac14@loongson.cn> Date: Wed, 14 Jun 2023 10:22:45 +0800 User-Agent: Mozilla/5.0 (X11; Linux loongarch64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: <38f4bfd4-fced-bd0b-6261-1b34cd894dda@amd.com> Content-Type: multipart/alternative; boundary="------------39BCED95E1386C58294F8568" Content-Language: en-US X-CM-TRANSID: AQAAf8DxVeR1JIlkCwgaAA--.9178S3 X-CM-SenderInfo: 52kr3ztlq6z05rqj20fqof0/ X-Coremail-Antispam: 1Uk129KBj93XoWxXF13Zw1fXw4xGw4DJr17XFc_yoWrtrWxpF W8GF1UJ3yqqr48Jw48ZFWrZry2vFs3J34UGry8Ga4UAr1DKry0qrn2gF1j9F4UArWvgr1j v3W7WF1Uur1DAacCm3ZEXasCq-sJn29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7KY7ZEXa sCq-sGcSsGvfJ3UbIjqfuFe4nvWSU5nxnvy29KBjDU0xBIdaVrnRJUUUkEb4IE77IF4wAF F20E14v26r1j6r4UM7CY07I20VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r 1Y6r17M28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAF wI0_JFI_Gr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv67 AKxVWxJr0_GcWl84ACjcxK6I8E87Iv6xkF7I0E14v26F4UJVW0owAS0I0E0xvYzxvE52x0 82IY62kv0487Mc804VCY07AIYIkI8VC2zVCFFI0UMcIj6xIIjxv20xvE14v26r106r15Mc Ij6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IY64vIr41l7480 Y4vEI4kI2Ix0rVAqx4xJMxk0xIA0c2IEe2xFo4CEbIxvr21l42xK82IYc2Ij64vIr41l4I 8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUGVWUWwC20s026x8GjcxK67AK xVWUGVWUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcV AFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r4UMIIF0xvE42xK8VAvwI8I cIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r 1j6r4UYxBIdaVFxhVjvjDU0xZFpf9x07UAWrXUUUUU= X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This is a multi-part message in MIME format. --------------39BCED95E1386C58294F8568 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hi Ferruh, On Wed, June 7, 2023 at 2:09AM, Ferruh Yigit wrote: > On 5/8/2023 2:21 AM, zhoumin wrote: >> Kindly ping. >> > Hi Min, > > Sorry for the delay. > >> Should we solve this problem by modifying PMDs? >> >> Is it suitable to maintain the RSS configurations for NIC by testpmd in >> order to set DCB? >> >> >> On Thur, Apr 27, 2023 at 4:33PM, zhoumin wrote: >>> Kindly ping. >>> >>> Any comments or suggestions will be appreciated. >>> >>> Best regards >>> Min >>> >>> >>> On 2023/4/12 下午5:52, Min Zhou wrote: >>>> In the testpmd command, we have to stop the port firstly before >>>> configuring >>>> the DCB. However, some PMDs may execute a hardware reset during the port >>>> stop, such as ixgbe. Some kind of reset operations of PMD could clear >>>> the >>>> configurations of RSS in the hardware register. This would cause the >>>> loss >>>> of RSS configurations that were set during the testpmd >>>> initialization. As >>>> a result, I find that I cannot enable RSS and DCB at the same time in >>>> the >>>> testpmd command when using Intel 82599 NIC. >>>> > cc'ed ixgbe maintainers too. OK, thanks. > > @Qiming, @Wenjun, are you testing DCB with RSS configuration? > > > Min, > > Can you please give more detail on the problem, do you have in only on > device reset, or is there a generic problem on enabling DCB with RSS? Yes, I will give more details on the problem blow. I think it is not a generic problem on enabling DCB with RSS. It depends on the implementation of "port stop" in the driver's codes. I can successfully enable DCB with RSS in the testpmd when it uses the i40e driver. It's very easy to reproduce this problem. I will demonstrate the process below: 1. First we start the testpmd program with enabling  RSS using 82599 NICs: $ ./build/app/dpdk-testpmd -l 0-6 -- -i --rss-ip --rxq=6 --txq=6 --nb-cores=6 2. Then show the RSS configuration in the testpmd: testpmd> show port 0 rss-hash RSS functions:   ipv4  ipv6  ipv6-ex 3. The above outputs are normal and then we try to enable DCB: testpmd> port config 0 dcb vt off 4 pfc on Please stop port 0 first 4. So we have to stop the port before enabling DCB: testpmd> port stop 0 Stopping ports... Checking link statuses... ixgbe_dev_link_update_share(): Other link thread is running now! Done 5. However, we will find that the RSS was disabled after stopping the port: testpmd> show port 0 rss-hash RSS disabled RSS is enabled in the MRQC register of 82599. The MRQC register supplied in section 8.2.3.7.12 of the Intel 82599 datasheet. I added some outputs before and after port reset in the port stop process of ixgbe driver as follows: diff --git a/drivers/net/ixgbe/base/ixgbe_82599.c b/drivers/net/ixgbe/base/ixgbe_82599.c index c6e8b7e976..b13431a43f 100644 --- a/drivers/net/ixgbe/base/ixgbe_82599.c +++ b/drivers/net/ixgbe/base/ixgbe_82599.c @@ -1053,6 +1053,7 @@ s32 ixgbe_reset_hw_82599(struct ixgbe_hw *hw)                 if (link_up)                         ctrl = IXGBE_CTRL_RST;         } +       printf("%s: MRQC = 0x%x\n", __func__, IXGBE_READ_REG(hw, IXGBE_MRQC));         ctrl |= IXGBE_READ_REG(hw, IXGBE_CTRL);         IXGBE_WRITE_REG(hw, IXGBE_CTRL, ctrl); @@ -1065,6 +1066,7 @@ s32 ixgbe_reset_hw_82599(struct ixgbe_hw *hw)                 if (!(ctrl & IXGBE_CTRL_RST_MASK))                         break;         } +       printf("%s: MRQC = 0x%x\n", __func__, IXGBE_READ_REG(hw, IXGBE_MRQC));         if (ctrl & IXGBE_CTRL_RST_MASK) {                 status = IXGBE_ERR_RESET_FAILED; We could find that the RSS configurations will be cleared after port reset as follows: testpmd> port stop 0 Stopping ports... ixgbe_reset_hw_82599: MRQC = 0x1a0001 ixgbe_reset_hw_82599: MRQC = 0x0 Checking link statuses... ixgbe_dev_link_update_share(): Other link thread is running now! Done However, I have not find the exact descriptions about this behavior in the 82599 datasheet. Could @Qiming, @Wenjun give me some indications ? So, if we want to get the RSS configurations by reading from the register after the port stop, we will get nothing. > Also can you please describe how/why removing RSS configuration (this > patch) helps on the problem? As above said, I'm concerned whether we could reuse the RSS configurations recorded in the testpmd if the port reset cannot be avoided in the driver. So, this patch wants to reuse the RSS configurations maintained by testpmd instead of reading from the register. I'm not sure if this is a reasonable method to solve this problem. > Thanks, > ferruh [...] Best regards, Min --------------39BCED95E1386C58294F8568 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

Hi Ferruh,

On Wed, June 7, 2023 at 2:09AM, Ferruh Yigit wrote:
On 5/8/2023 2:21 AM, zhoumin wrote:

Kindly ping.


Hi Min,

Sorry for the delay.


Should we solve this problem by modifying PMDs?

Is it suitable to maintain the RSS configurations for NIC by testpmd in
order to set DCB?


On Thur, Apr 27, 2023 at 4:33PM, zhoumin wrote:

Kindly ping.

Any comments or suggestions will be appreciated.

Best regards
Min


On 2023/4/12 下午5:52, Min Zhou wrote:

In the testpmd command, we have to stop the port firstly before
configuring
the DCB. However, some PMDs may execute a hardware reset during the port
stop, such as ixgbe. Some kind of reset operations of PMD could clear
the
configurations of RSS in the hardware register. This would cause the
loss
of RSS configurations that were set during the testpmd
initialization. As
a result, I find that I cannot enable RSS and DCB at the same time in
the
testpmd command when using Intel 82599 NIC.


cc'ed ixgbe maintainers too.
OK, thanks.

@Qiming, @Wenjun, are you testing DCB with RSS configuration?


Min,

Can you please give more detail on the problem, do you have in only on
device reset, or is there a generic problem on enabling DCB with RSS?

Yes, I will give more details on the problem blow.

I think it is not a generic problem on enabling DCB with RSS. It depends on the implementation of "port stop" in the driver's codes. I can successfully enable DCB with RSS in the testpmd when it uses the i40e driver.

It's very easy to reproduce this problem. I will demonstrate the process below:

1. First we start the testpmd program with enabling  RSS using 82599 NICs:

$ ./build/app/dpdk-testpmd -l 0-6 -- -i --rss-ip --rxq=6 --txq=6 --nb-cores=6

2. Then show the RSS configuration in the testpmd:

testpmd> show port 0 rss-hash

RSS functions:
  ipv4  ipv6  ipv6-ex

3. The above outputs are normal and then we try to enable DCB:

testpmd> port config 0 dcb vt off 4 pfc on
Please stop port 0 first

4. So we have to stop the port before enabling DCB:

testpmd> port stop 0
Stopping ports...
Checking link statuses...
ixgbe_dev_link_update_share(): Other link thread is running now!
Done

5. However, we will find that the RSS was disabled after stopping the port:

testpmd> show port 0 rss-hash
RSS disabled


RSS is enabled in the MRQC register of 82599. The MRQC register supplied in section 8.2.3.7.12 of the Intel 82599 datasheet.

I added some outputs before and after port reset in the port stop process of ixgbe driver as follows:

diff --git a/drivers/net/ixgbe/base/ixgbe_82599.c b/drivers/net/ixgbe/base/ixgbe_82599.c
index c6e8b7e976..b13431a43f 100644
--- a/drivers/net/ixgbe/base/ixgbe_82599.c
+++ b/drivers/net/ixgbe/base/ixgbe_82599.c
@@ -1053,6 +1053,7 @@ s32 ixgbe_reset_hw_82599(struct ixgbe_hw *hw)
                if (link_up)
                        ctrl = IXGBE_CTRL_RST;
        }
+       printf("%s: MRQC = 0x%x\n", __func__, IXGBE_READ_REG(hw, IXGBE_MRQC));
 
        ctrl |= IXGBE_READ_REG(hw, IXGBE_CTRL);
        IXGBE_WRITE_REG(hw, IXGBE_CTRL, ctrl);
@@ -1065,6 +1066,7 @@ s32 ixgbe_reset_hw_82599(struct ixgbe_hw *hw)
                if (!(ctrl & IXGBE_CTRL_RST_MASK))
                        break;
        }
+       printf("%s: MRQC = 0x%x\n", __func__, IXGBE_READ_REG(hw, IXGBE_MRQC));
 
        if (ctrl & IXGBE_CTRL_RST_MASK) {
                status = IXGBE_ERR_RESET_FAILED;

We could find that the RSS configurations will be cleared after port reset as follows:

testpmd> port stop 0
Stopping ports...
ixgbe_reset_hw_82599: MRQC = 0x1a0001
ixgbe_reset_hw_82599: MRQC = 0x0
Checking link statuses...
ixgbe_dev_link_update_share(): Other link thread is running now!
Done

However, I have not find the exact descriptions about this behavior in the 82599 datasheet. Could @Qiming, @Wenjun give me some indications ?

So, if we want to get the RSS configurations by reading from the register after the port stop, we will get nothing.

Also can you please describe how/why removing RSS configuration (this
patch) helps on the problem?

As above said, I'm concerned whether we could reuse the RSS configurations recorded in the testpmd if the port reset cannot be avoided in the driver.

So, this patch wants to reuse the RSS configurations maintained by testpmd instead of reading from the register.

I'm not sure if this is a reasonable method to solve this problem.

Thanks,
ferruh

[...]

Best regards,

Min


--------------39BCED95E1386C58294F8568--