[PATCH] net/nfp: set the appropriate initialized value of flbufsz

[PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Chaoyong He]

From: Peng Zhang <peng.zhang@corigine.com>

When the testpmd app start-up with parameter max-pkt-len, it will set MTU.
But the initialized value of flubfsz is inappropriate, if the value of
flbufsz is smaller than the valude of max-pkt-len, the testpmd app will
start fail.

Fixes: 5c305e218f15 ("net/nfp: fix initialization")
Cc: stable@dpdk.org

Signed-off-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
---
 drivers/net/nfp/nfp_ethdev.c    | 2 +-
 drivers/net/nfp/nfp_ethdev_vf.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/nfp/nfp_ethdev.c b/drivers/net/nfp/nfp_ethdev.c
index 5cdd34e588..b95e623f1f 100644
--- a/drivers/net/nfp/nfp_ethdev.c
+++ b/drivers/net/nfp/nfp_ethdev.c
@@ -517,7 +517,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev)
 	hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
 	hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
 	hw->mtu = RTE_ETHER_MTU;
-	hw->flbufsz = RTE_ETHER_MTU;
+	hw->flbufsz = hw->max_mtu;
 
 	/* VLAN insertion is incompatible with LSOv2 */
 	if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
diff --git a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
index d304d78d34..47acb4c60e 100644
--- a/drivers/net/nfp/nfp_ethdev_vf.c
+++ b/drivers/net/nfp/nfp_ethdev_vf.c
@@ -396,7 +396,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)
 	hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
 	hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
 	hw->mtu = RTE_ETHER_MTU;
-	hw->flbufsz = RTE_ETHER_MTU;
+	hw->flbufsz = hw->max_mtu;
 
 	/* VLAN insertion is incompatible with LSOv2 */
 	if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
-- 
2.27.0

Re: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Ferruh Yigit]

On 10/10/2022 7:48 AM, Chaoyong He wrote:
> From: Peng Zhang <peng.zhang@corigine.com>
> 
> When the testpmd app start-up with parameter max-pkt-len, it will set MTU.
> But the initialized value of flubfsz is inappropriate, if the value of
> flbufsz is smaller than the valude of max-pkt-len, the testpmd app will
> start fail.
> 

What is the failure in the testpmd?

This patch is fixing something but it is not clear what is fixed, the 
concern is it may be changing driver to make something pass in test 
application (testpmd).

What is 'flubfsz', is it Hw configured frame buffer size?


> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Peng Zhang <peng.zhang@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
> ---
>   drivers/net/nfp/nfp_ethdev.c    | 2 +-
>   drivers/net/nfp/nfp_ethdev_vf.c | 2 +-
>   2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/nfp/nfp_ethdev.c b/drivers/net/nfp/nfp_ethdev.c
> index 5cdd34e588..b95e623f1f 100644
> --- a/drivers/net/nfp/nfp_ethdev.c
> +++ b/drivers/net/nfp/nfp_ethdev.c
> @@ -517,7 +517,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev)
>   	hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
>   	hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
>   	hw->mtu = RTE_ETHER_MTU;
> -	hw->flbufsz = RTE_ETHER_MTU;
> +	hw->flbufsz = hw->max_mtu;
>   
>   	/* VLAN insertion is incompatible with LSOv2 */
>   	if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
> diff --git a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
> index d304d78d34..47acb4c60e 100644
> --- a/drivers/net/nfp/nfp_ethdev_vf.c
> +++ b/drivers/net/nfp/nfp_ethdev_vf.c
> @@ -396,7 +396,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)
>   	hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
>   	hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
>   	hw->mtu = RTE_ETHER_MTU;
> -	hw->flbufsz = RTE_ETHER_MTU;
> +	hw->flbufsz = hw->max_mtu;
>   
>   	/* VLAN insertion is incompatible with LSOv2 */
>   	if (hw->cap & NFP_NET_CFG_CTRL_LSO2)

RE: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Nole Zhang]

 > On 10/10/2022 7:48 AM, Chaoyong He wrote:
> > From: Peng Zhang <peng.zhang@corigine.com>
> >
> > When the testpmd app start-up with parameter max-pkt-len, it will set
> MTU.
> > But the initialized value of flubfsz is inappropriate, if the value of
> > flbufsz is smaller than the valude of max-pkt-len, the testpmd app
> > will start fail.
> >
> 
> What is the failure in the testpmd?

The log is as follows:
[root@volstruis ~]# dpdk-testpmd --main-lcore 10 -l 10,11,12 -n 4 -a 0000:81:00.0 --socket-mem 2048,2048 --proc-type auto -- --portmask 0x3 --nb-cores 2 --rxq 1 --txq 1 --rxd 1024 --txd 1024 --port-topology loop --forward-mode macswap  --max-pkt-len 9216 --mbuf-size 9600  --rss-udp --burst=32
EAL: Detected CPU lcores: 40
EAL: Detected NUMA nodes: 2
EAL: Auto-detected process type: PRIMARY
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_nfp_pf (19ee:4000) device: 0000:81:00.0 (socket 1) NFP HWINFO header: 48490200
TELEMETRY: No legacy callbacks, legacy socket not created Set macswap packet forwarding mode
testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=9600, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1)
Port0 dev_configure = -34
Fail to configure port 0
EAL: Error - exiting with code: 1
  Cause: Start ports failed

First  in the `nfp_net_configure()`, we will judge the value of MTU and hw->flbufsz, If MTU > hw->flbufsz, it will have the error.

And the `--max-pkt-len` is setting the MTU in the initialize process, the initialized  value of  hw->flbufsz is just 1500 at first.

So if we set the `max-pkt-len`  bigger than the initialized value of flbufsz, It will lead the error.

Hence we set the new value of hw->flbufsz, it can large the range max-pkt-len in the initialized process.


> 
> This patch is fixing something but it is not clear what is fixed, the concern is it
> may be changing driver to make something pass in test application (testpmd).
> 
> What is 'flubfsz', is it Hw configured frame buffer size?


It is configured in the `nfp_net_rx_queue_setup()`{`hw->flbufsz = rxq->mbuf_size`}.
If the rxq->mbuf_size < MTU, the MTU can't work.

> 
> 
> > Fixes: 5c305e218f15 ("net/nfp: fix initialization")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Peng Zhang <peng.zhang@corigine.com>
> > Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> > Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
> > ---
> >   drivers/net/nfp/nfp_ethdev.c    | 2 +-
> >   drivers/net/nfp/nfp_ethdev_vf.c | 2 +-
> >   2 files changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/net/nfp/nfp_ethdev.c
> > b/drivers/net/nfp/nfp_ethdev.c index 5cdd34e588..b95e623f1f 100644
> > --- a/drivers/net/nfp/nfp_ethdev.c
> > +++ b/drivers/net/nfp/nfp_ethdev.c
> > @@ -517,7 +517,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev)
> >       hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
> >       hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
> >       hw->mtu = RTE_ETHER_MTU;
> > -     hw->flbufsz = RTE_ETHER_MTU;
> > +     hw->flbufsz = hw->max_mtu;
> >
> >       /* VLAN insertion is incompatible with LSOv2 */
> >       if (hw->cap & NFP_NET_CFG_CTRL_LSO2) diff --git
> > a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
> > index d304d78d34..47acb4c60e 100644
> > --- a/drivers/net/nfp/nfp_ethdev_vf.c
> > +++ b/drivers/net/nfp/nfp_ethdev_vf.c
> > @@ -396,7 +396,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)
> >       hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
> >       hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
> >       hw->mtu = RTE_ETHER_MTU;
> > -     hw->flbufsz = RTE_ETHER_MTU;
> > +     hw->flbufsz = hw->max_mtu;
> >
> >       /* VLAN insertion is incompatible with LSOv2 */
> >       if (hw->cap & NFP_NET_CFG_CTRL_LSO2)

Re: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Ferruh Yigit]

On 10/15/2022 8:38 AM, Nole Zhang wrote:
>   > On 10/10/2022 7:48 AM, Chaoyong He wrote:
>>> From: Peng Zhang <peng.zhang@corigine.com>
>>>
>>> When the testpmd app start-up with parameter max-pkt-len, it will set
>> MTU.
>>> But the initialized value of flubfsz is inappropriate, if the value of
>>> flbufsz is smaller than the valude of max-pkt-len, the testpmd app
>>> will start fail.
>>>
>>
>> What is the failure in the testpmd?
> 
> The log is as follows:
> [root@volstruis ~]# dpdk-testpmd --main-lcore 10 -l 10,11,12 -n 4 -a 0000:81:00.0 --socket-mem 2048,2048 --proc-type auto -- --portmask 0x3 --nb-cores 2 --rxq 1 --txq 1 --rxd 1024 --txd 1024 --port-topology loop --forward-mode macswap  --max-pkt-len 9216 --mbuf-size 9600  --rss-udp --burst=32
> EAL: Detected CPU lcores: 40
> EAL: Detected NUMA nodes: 2
> EAL: Auto-detected process type: PRIMARY
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> EAL: Using IOMMU type 1 (Type 1)
> EAL: Probe PCI driver: net_nfp_pf (19ee:4000) device: 0000:81:00.0 (socket 1) NFP HWINFO header: 48490200
> TELEMETRY: No legacy callbacks, legacy socket not created Set macswap packet forwarding mode
> testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=9600, socket=1
> testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1)
> Port0 dev_configure = -34
> Fail to configure port 0
> EAL: Error - exiting with code: 1
>    Cause: Start ports failed
> 
> First  in the `nfp_net_configure()`, we will judge the value of MTU and hw->flbufsz, If MTU > hw->flbufsz, it will have the error.
> 
> And the `--max-pkt-len` is setting the MTU in the initialize process, the initialized  value of  hw->flbufsz is just 1500 at first.
> 
> So if we set the `max-pkt-len`  bigger than the initialized value of flbufsz, It will lead the error.
> 
> Hence we set the new value of hw->flbufsz, it can large the range max-pkt-len in the initialized process.
> 
> 
>>
>> This patch is fixing something but it is not clear what is fixed, the concern is it
>> may be changing driver to make something pass in test application (testpmd).
>>
>> What is 'flubfsz', is it Hw configured frame buffer size?
> 
> 
> It is configured in the `nfp_net_rx_queue_setup()`{`hw->flbufsz = rxq->mbuf_size`}.
> If the rxq->mbuf_size < MTU, the MTU can't work.
> 

It looks like `hw->flbufsz` holds the Rx buffer size, as you highlighted 
above.

And you don't want to accept frames bigger than buffer size, since it 
seems driver doesn't support `RTE_ETH_RX_OFFLOAD_SCATTER`, all looks OK.


According above logic, I agree "hw->flbufsz = RTE_ETHER_MTU;" is wrong, 
but equally `hw->flbufsz = hw->max_mtu;` seems wrong.

In above command line, it is safe because "mbuf-size=9600" and 
"max-pkt-len=9216", buffer size is bigger than packet size.

You should able to set `hw->flbufsz` to current buffer size, instead of 
a hardcoded value.

In `nfp_net_init()`, most probably you don't know the buffer size yet, 
can't you skip setting this value here and set it in 
`nfp_net_rx_queue_setup()` when you know the buffer size?


>>
>>
>>> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
>>> Cc: stable@dpdk.org
>>>
>>> Signed-off-by: Peng Zhang <peng.zhang@corigine.com>
>>> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
>>> Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
>>> ---
>>>    drivers/net/nfp/nfp_ethdev.c    | 2 +-
>>>    drivers/net/nfp/nfp_ethdev_vf.c | 2 +-
>>>    2 files changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/net/nfp/nfp_ethdev.c
>>> b/drivers/net/nfp/nfp_ethdev.c index 5cdd34e588..b95e623f1f 100644
>>> --- a/drivers/net/nfp/nfp_ethdev.c
>>> +++ b/drivers/net/nfp/nfp_ethdev.c
>>> @@ -517,7 +517,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev)
>>>        hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
>>>        hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
>>>        hw->mtu = RTE_ETHER_MTU;
>>> -     hw->flbufsz = RTE_ETHER_MTU;
>>> +     hw->flbufsz = hw->max_mtu;
>>>
>>>        /* VLAN insertion is incompatible with LSOv2 */
>>>        if (hw->cap & NFP_NET_CFG_CTRL_LSO2) diff --git
>>> a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
>>> index d304d78d34..47acb4c60e 100644
>>> --- a/drivers/net/nfp/nfp_ethdev_vf.c
>>> +++ b/drivers/net/nfp/nfp_ethdev_vf.c
>>> @@ -396,7 +396,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)
>>>        hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
>>>        hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
>>>        hw->mtu = RTE_ETHER_MTU;
>>> -     hw->flbufsz = RTE_ETHER_MTU;
>>> +     hw->flbufsz = hw->max_mtu;
>>>
>>>        /* VLAN insertion is incompatible with LSOv2 */
>>>        if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
> 

RE: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Nole Zhang]

> On 10/15/2022 8:38 AM, Nole Zhang wrote:
> >   > On 10/10/2022 7:48 AM, Chaoyong He wrote:
> >>> From: Peng Zhang <peng.zhang@corigine.com>
> >>>
> >>> When the testpmd app start-up with parameter max-pkt-len, it will
> >>> set
> >> MTU.
> >>> But the initialized value of flubfsz is inappropriate, if the value
> >>> of flbufsz is smaller than the valude of max-pkt-len, the testpmd
> >>> app will start fail.
> >>>
> >>
> >> What is the failure in the testpmd?
> >
> > The log is as follows:
> > [root@volstruis ~]# dpdk-testpmd --main-lcore 10 -l 10,11,12 -n 4 -a
> > 0000:81:00.0 --socket-mem 2048,2048 --proc-type auto -- --portmask 0x3
> > --nb-cores 2 --rxq 1 --txq 1 --rxd 1024 --txd 1024 --port-topology
> > loop --forward-mode macswap  --max-pkt-len 9216 --mbuf-size 9600
> > --rss-udp --burst=32
> > EAL: Detected CPU lcores: 40
> > EAL: Detected NUMA nodes: 2
> > EAL: Auto-detected process type: PRIMARY
> > EAL: Detected static linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'VA'
> > EAL: VFIO support initialized
> > EAL: Using IOMMU type 1 (Type 1)
> > EAL: Probe PCI driver: net_nfp_pf (19ee:4000) device: 0000:81:00.0
> > (socket 1) NFP HWINFO header: 48490200
> > TELEMETRY: No legacy callbacks, legacy socket not created Set macswap
> > packet forwarding mode
> > testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=9600,
> > socket=1
> > testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0
> > (socket 1)
> > Port0 dev_configure = -34
> > Fail to configure port 0
> > EAL: Error - exiting with code: 1
> >    Cause: Start ports failed
> >
> > First  in the `nfp_net_configure()`, we will judge the value of MTU and hw-
> >flbufsz, If MTU > hw->flbufsz, it will have the error.
> >
> > And the `--max-pkt-len` is setting the MTU in the initialize process, the
> initialized  value of  hw->flbufsz is just 1500 at first.
> >
> > So if we set the `max-pkt-len`  bigger than the initialized value of flbufsz, It
> will lead the error.
> >
> > Hence we set the new value of hw->flbufsz, it can large the range max-pkt-
> len in the initialized process.
> >
> >
> >>
> >> This patch is fixing something but it is not clear what is fixed, the
> >> concern is it may be changing driver to make something pass in test
> application (testpmd).
> >>
> >> What is 'flubfsz', is it Hw configured frame buffer size?
> >
> >
> > It is configured in the `nfp_net_rx_queue_setup()`{`hw->flbufsz = rxq-
> >mbuf_size`}.
> > If the rxq->mbuf_size < MTU, the MTU can't work.
> >
> 
> It looks like `hw->flbufsz` holds the Rx buffer size, as you highlighted above.
> 
> And you don't want to accept frames bigger than buffer size, since it seems
> driver doesn't support `RTE_ETH_RX_OFFLOAD_SCATTER`, all looks OK.
> 
> 
> According above logic, I agree "hw->flbufsz = RTE_ETHER_MTU;" is wrong,
> but equally `hw->flbufsz = hw->max_mtu;` seems wrong.
> 
> In above command line, it is safe because "mbuf-size=9600" and
> "max-pkt-len=9216", buffer size is bigger than packet size.
Yes, if I need set the hardcoded value, I should set the max max-pkt-len.
> 
> You should able to set `hw->flbufsz` to current buffer size, instead of
> a hardcoded value.
> 
> In `nfp_net_init()`, most probably you don't know the buffer size yet,
> can't you skip setting this value here and set it in
> `nfp_net_rx_queue_setup()` when you know the buffer size?
> 

But If I just depends on the `nfp_net_rx_queue_setup()`, in the `nfp_net_init()`, it will
Call the  `nfp_net_configure()`, it will lead the testpmd start failed, so I add the hardcoded value
in the initialize process. Or  I can remove the judge about  `hw->flbufsz` in the `nfp_net_init()`.

Thanks for your advice.
> 
> >>
> >>
> >>> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
> >>> Cc: stable@dpdk.org
> >>>
> >>> Signed-off-by: Peng Zhang <peng.zhang@corigine.com>
> >>> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> >>> Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
> >>> ---
> >>>    drivers/net/nfp/nfp_ethdev.c    | 2 +-
> >>>    drivers/net/nfp/nfp_ethdev_vf.c | 2 +-
> >>>    2 files changed, 2 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/drivers/net/nfp/nfp_ethdev.c
> >>> b/drivers/net/nfp/nfp_ethdev.c index 5cdd34e588..b95e623f1f 100644
> >>> --- a/drivers/net/nfp/nfp_ethdev.c
> >>> +++ b/drivers/net/nfp/nfp_ethdev.c
> >>> @@ -517,7 +517,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev)
> >>>        hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
> >>>        hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
> >>>        hw->mtu = RTE_ETHER_MTU;
> >>> -     hw->flbufsz = RTE_ETHER_MTU;
> >>> +     hw->flbufsz = hw->max_mtu;
> >>>
> >>>        /* VLAN insertion is incompatible with LSOv2 */
> >>>        if (hw->cap & NFP_NET_CFG_CTRL_LSO2) diff --git
> >>> a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
> >>> index d304d78d34..47acb4c60e 100644
> >>> --- a/drivers/net/nfp/nfp_ethdev_vf.c
> >>> +++ b/drivers/net/nfp/nfp_ethdev_vf.c
> >>> @@ -396,7 +396,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)
> >>>        hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
> >>>        hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
> >>>        hw->mtu = RTE_ETHER_MTU;
> >>> -     hw->flbufsz = RTE_ETHER_MTU;
> >>> +     hw->flbufsz = hw->max_mtu;
> >>>
> >>>        /* VLAN insertion is incompatible with LSOv2 */
> >>>        if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
> >

Re: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Ferruh Yigit]

On 10/18/2022 2:41 AM, Nole Zhang wrote:
> 
>> On 10/15/2022 8:38 AM, Nole Zhang wrote:
>>>    > On 10/10/2022 7:48 AM, Chaoyong He wrote:
>>>>> From: Peng Zhang <peng.zhang@corigine.com>
>>>>>
>>>>> When the testpmd app start-up with parameter max-pkt-len, it will
>>>>> set
>>>> MTU.
>>>>> But the initialized value of flubfsz is inappropriate, if the value
>>>>> of flbufsz is smaller than the valude of max-pkt-len, the testpmd
>>>>> app will start fail.
>>>>>
>>>>
>>>> What is the failure in the testpmd?
>>>
>>> The log is as follows:
>>> [root@volstruis ~]# dpdk-testpmd --main-lcore 10 -l 10,11,12 -n 4 -a
>>> 0000:81:00.0 --socket-mem 2048,2048 --proc-type auto -- --portmask 0x3
>>> --nb-cores 2 --rxq 1 --txq 1 --rxd 1024 --txd 1024 --port-topology
>>> loop --forward-mode macswap  --max-pkt-len 9216 --mbuf-size 9600
>>> --rss-udp --burst=32
>>> EAL: Detected CPU lcores: 40
>>> EAL: Detected NUMA nodes: 2
>>> EAL: Auto-detected process type: PRIMARY
>>> EAL: Detected static linkage of DPDK
>>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>>> EAL: Selected IOVA mode 'VA'
>>> EAL: VFIO support initialized
>>> EAL: Using IOMMU type 1 (Type 1)
>>> EAL: Probe PCI driver: net_nfp_pf (19ee:4000) device: 0000:81:00.0
>>> (socket 1) NFP HWINFO header: 48490200
>>> TELEMETRY: No legacy callbacks, legacy socket not created Set macswap
>>> packet forwarding mode
>>> testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=9600,
>>> socket=1
>>> testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0
>>> (socket 1)
>>> Port0 dev_configure = -34
>>> Fail to configure port 0
>>> EAL: Error - exiting with code: 1
>>>     Cause: Start ports failed
>>>
>>> First  in the `nfp_net_configure()`, we will judge the value of MTU and hw-
>>> flbufsz, If MTU > hw->flbufsz, it will have the error.
>>>
>>> And the `--max-pkt-len` is setting the MTU in the initialize process, the
>> initialized  value of  hw->flbufsz is just 1500 at first.
>>>
>>> So if we set the `max-pkt-len`  bigger than the initialized value of flbufsz, It
>> will lead the error.
>>>
>>> Hence we set the new value of hw->flbufsz, it can large the range max-pkt-
>> len in the initialized process.
>>>
>>>
>>>>
>>>> This patch is fixing something but it is not clear what is fixed, the
>>>> concern is it may be changing driver to make something pass in test
>> application (testpmd).
>>>>
>>>> What is 'flubfsz', is it Hw configured frame buffer size?
>>>
>>>
>>> It is configured in the `nfp_net_rx_queue_setup()`{`hw->flbufsz = rxq-
>>> mbuf_size`}.
>>> If the rxq->mbuf_size < MTU, the MTU can't work.
>>>
>>
>> It looks like `hw->flbufsz` holds the Rx buffer size, as you highlighted above.
>>
>> And you don't want to accept frames bigger than buffer size, since it seems
>> driver doesn't support `RTE_ETH_RX_OFFLOAD_SCATTER`, all looks OK.
>>
>>
>> According above logic, I agree "hw->flbufsz = RTE_ETHER_MTU;" is wrong,
>> but equally `hw->flbufsz = hw->max_mtu;` seems wrong.
>>
>> In above command line, it is safe because "mbuf-size=9600" and
>> "max-pkt-len=9216", buffer size is bigger than packet size.
> Yes, if I need set the hardcoded value, I should set the max max-pkt-len.

I think it should be 'mbuf-size', since 'hw->flbufsz' is the buffer size.
'max-pkt-len' is the max Rx frame accepted by NIC, for some devices 
'max-pkt-len' can be bigger than buffer size but it will work fine 
because of segmented Rx, but it seems this config doesn't work for nfp.

>>
>> You should able to set `hw->flbufsz` to current buffer size, instead of
>> a hardcoded value.
>>
>> In `nfp_net_init()`, most probably you don't know the buffer size yet,
>> can't you skip setting this value here and set it in
>> `nfp_net_rx_queue_setup()` when you know the buffer size?
>>
> 
> But If I just depends on the `nfp_net_rx_queue_setup()`, in the `nfp_net_init()`, it will
> Call the  `nfp_net_configure()`, it will lead the testpmd start failed, so I add the hardcoded value
> in the initialize process. Or  I can remove the judge about  `hw->flbufsz` in the `nfp_net_init()`.
> 

Instead of setting 'hw->flbufsz' in 'nfp_net_init()' and check it in 
'nfp_net_configure()', why not
set 'hw->flbufsz' in 'nfp_net_configure()'?

Because in 'nfp_net_configure()', 'mtu' is a configuration parameter. It 
can be possible to convert 'mtu' to frame size and set it. If there is a 
HW limitation on buffer size, it can fail in 'nfp_net_configure()' when 
that limit hit.

> Thanks for your advice.
>>
>>>>
>>>>
>>>>> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
>>>>> Cc: stable@dpdk.org
>>>>>
>>>>> Signed-off-by: Peng Zhang <peng.zhang@corigine.com>
>>>>> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
>>>>> Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
>>>>> ---
>>>>>     drivers/net/nfp/nfp_ethdev.c    | 2 +-
>>>>>     drivers/net/nfp/nfp_ethdev_vf.c | 2 +-
>>>>>     2 files changed, 2 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/drivers/net/nfp/nfp_ethdev.c
>>>>> b/drivers/net/nfp/nfp_ethdev.c index 5cdd34e588..b95e623f1f 100644
>>>>> --- a/drivers/net/nfp/nfp_ethdev.c
>>>>> +++ b/drivers/net/nfp/nfp_ethdev.c
>>>>> @@ -517,7 +517,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev)
>>>>>         hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
>>>>>         hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
>>>>>         hw->mtu = RTE_ETHER_MTU;
>>>>> -     hw->flbufsz = RTE_ETHER_MTU;
>>>>> +     hw->flbufsz = hw->max_mtu;
>>>>>
>>>>>         /* VLAN insertion is incompatible with LSOv2 */
>>>>>         if (hw->cap & NFP_NET_CFG_CTRL_LSO2) diff --git
>>>>> a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
>>>>> index d304d78d34..47acb4c60e 100644
>>>>> --- a/drivers/net/nfp/nfp_ethdev_vf.c
>>>>> +++ b/drivers/net/nfp/nfp_ethdev_vf.c
>>>>> @@ -396,7 +396,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)
>>>>>         hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
>>>>>         hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
>>>>>         hw->mtu = RTE_ETHER_MTU;
>>>>> -     hw->flbufsz = RTE_ETHER_MTU;
>>>>> +     hw->flbufsz = hw->max_mtu;
>>>>>
>>>>>         /* VLAN insertion is incompatible with LSOv2 */
>>>>>         if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
>>>
> 

RE: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Nole Zhang]

> On 10/18/2022 2:41 AM, Nole Zhang wrote:
> >
> >> On 10/15/2022 8:38 AM, Nole Zhang wrote:
> >>>    > On 10/10/2022 7:48 AM, Chaoyong He wrote:
> >>>>> From: Peng Zhang <peng.zhang@corigine.com>
> >>>>>
> >>>>> When the testpmd app start-up with parameter max-pkt-len, it will
> >>>>> set
> >>>> MTU.
> >>>>> But the initialized value of flubfsz is inappropriate, if the
> >>>>> value of flbufsz is smaller than the valude of max-pkt-len, the
> >>>>> testpmd app will start fail.
> >>>>>
> >>>>
> >>>> What is the failure in the testpmd?
> >>>
> >>> The log is as follows:
> >>> [root@volstruis ~]# dpdk-testpmd --main-lcore 10 -l 10,11,12 -n 4 -a
> >>> 0000:81:00.0 --socket-mem 2048,2048 --proc-type auto -- --portmask
> >>> 0x3 --nb-cores 2 --rxq 1 --txq 1 --rxd 1024 --txd 1024
> >>> --port-topology loop --forward-mode macswap  --max-pkt-len 9216
> >>> --mbuf-size 9600 --rss-udp --burst=32
> >>> EAL: Detected CPU lcores: 40
> >>> EAL: Detected NUMA nodes: 2
> >>> EAL: Auto-detected process type: PRIMARY
> >>> EAL: Detected static linkage of DPDK
> >>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >>> EAL: Selected IOVA mode 'VA'
> >>> EAL: VFIO support initialized
> >>> EAL: Using IOMMU type 1 (Type 1)
> >>> EAL: Probe PCI driver: net_nfp_pf (19ee:4000) device: 0000:81:00.0
> >>> (socket 1) NFP HWINFO header: 48490200
> >>> TELEMETRY: No legacy callbacks, legacy socket not created Set
> >>> macswap packet forwarding mode
> >>> testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=9600,
> >>> socket=1
> >>> testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port
> >>> 0 (socket 1)
> >>> Port0 dev_configure = -34
> >>> Fail to configure port 0
> >>> EAL: Error - exiting with code: 1
> >>>     Cause: Start ports failed
> >>>
> >>> First  in the `nfp_net_configure()`, we will judge the value of MTU
> >>> and hw- flbufsz, If MTU > hw->flbufsz, it will have the error.
> >>>
> >>> And the `--max-pkt-len` is setting the MTU in the initialize
> >>> process, the
> >> initialized  value of  hw->flbufsz is just 1500 at first.
> >>>
> >>> So if we set the `max-pkt-len`  bigger than the initialized value of
> >>> flbufsz, It
> >> will lead the error.
> >>>
> >>> Hence we set the new value of hw->flbufsz, it can large the range
> >>> max-pkt-
> >> len in the initialized process.
> >>>
> >>>
> >>>>
> >>>> This patch is fixing something but it is not clear what is fixed,
> >>>> the concern is it may be changing driver to make something pass in
> >>>> test
> >> application (testpmd).
> >>>>
> >>>> What is 'flubfsz', is it Hw configured frame buffer size?
> >>>
> >>>
> >>> It is configured in the `nfp_net_rx_queue_setup()`{`hw->flbufsz =
> >>> rxq- mbuf_size`}.
> >>> If the rxq->mbuf_size < MTU, the MTU can't work.
> >>>
> >>
> >> It looks like `hw->flbufsz` holds the Rx buffer size, as you highlighted
> above.
> >>
> >> And you don't want to accept frames bigger than buffer size, since it
> >> seems driver doesn't support `RTE_ETH_RX_OFFLOAD_SCATTER`, all looks
> OK.
> >>
> >>
> >> According above logic, I agree "hw->flbufsz = RTE_ETHER_MTU;" is
> >> wrong, but equally `hw->flbufsz = hw->max_mtu;` seems wrong.
> >>
> >> In above command line, it is safe because "mbuf-size=9600" and
> >> "max-pkt-len=9216", buffer size is bigger than packet size.
> > Yes, if I need set the hardcoded value, I should set the max max-pkt-len.
> 
> I think it should be 'mbuf-size', since 'hw->flbufsz' is the buffer size.
> 'max-pkt-len' is the max Rx frame accepted by NIC, for some devices 'max-
> pkt-len' can be bigger than buffer size but it will work fine because of
> segmented Rx, but it seems this config doesn't work for nfp.
> 
As you said, maybe the ` NFP_FRAME_SIZE_MAX ` is right, for our nic,
the max_rx_pktlen is NFP_FRAME_SIZE_MAX.
> >>
> >> You should able to set `hw->flbufsz` to current buffer size, instead
> >> of a hardcoded value.
> >>
> >> In `nfp_net_init()`, most probably you don't know the buffer size
> >> yet, can't you skip setting this value here and set it in
> >> `nfp_net_rx_queue_setup()` when you know the buffer size?
> >>
> >
> > But If I just depends on the `nfp_net_rx_queue_setup()`, in the
> > `nfp_net_init()`, it will Call the  `nfp_net_configure()`, it will
> > lead the testpmd start failed, so I add the hardcoded value in the initialize
> process. Or  I can remove the judge about  `hw->flbufsz` in the
> `nfp_net_init()`.
> >
> 
> Instead of setting 'hw->flbufsz' in 'nfp_net_init()' and check it in
> 'nfp_net_configure()', why not set 'hw->flbufsz' in 'nfp_net_configure()'?
> 
> Because in 'nfp_net_configure()', 'mtu' is a configuration parameter. It can
> be possible to convert 'mtu' to frame size and set it. If there is a HW
> limitation on buffer size, it can fail in 'nfp_net_configure()' when that limit hit.
> 
You mean in the 'nfp_net_configure()', set the value of 'hw->flbufsz', 
Like 
`if (rxmode->mtu > hw->flbufsz) {
		hw->flbufsz = rxmode->mtu;
	}`
Because the 'nfp_net_configure()' also isn't only called once.
If it will be set the values every times, so I first just set the
Initial value in the `nfp_net_inital()`.

> > Thanks for your advice.
> >>
> >>>>
> >>>>
> >>>>> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
> >>>>> Cc: stable@dpdk.org
...
> >>>>>         /* VLAN insertion is incompatible with LSOv2 */
> >>>>>         if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
> >>>
> >

Re: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Ferruh Yigit]

On 10/18/2022 9:51 AM, Nole Zhang wrote:
>> On 10/18/2022 2:41 AM, Nole Zhang wrote:
>>>
>>>> On 10/15/2022 8:38 AM, Nole Zhang wrote:
>>>>>     > On 10/10/2022 7:48 AM, Chaoyong He wrote:
>>>>>>> From: Peng Zhang <peng.zhang@corigine.com>
>>>>>>>
>>>>>>> When the testpmd app start-up with parameter max-pkt-len, it will
>>>>>>> set
>>>>>> MTU.
>>>>>>> But the initialized value of flubfsz is inappropriate, if the
>>>>>>> value of flbufsz is smaller than the valude of max-pkt-len, the
>>>>>>> testpmd app will start fail.
>>>>>>>
>>>>>>
>>>>>> What is the failure in the testpmd?
>>>>>
>>>>> The log is as follows:
>>>>> [root@volstruis ~]# dpdk-testpmd --main-lcore 10 -l 10,11,12 -n 4 -a
>>>>> 0000:81:00.0 --socket-mem 2048,2048 --proc-type auto -- --portmask
>>>>> 0x3 --nb-cores 2 --rxq 1 --txq 1 --rxd 1024 --txd 1024
>>>>> --port-topology loop --forward-mode macswap  --max-pkt-len 9216
>>>>> --mbuf-size 9600 --rss-udp --burst=32
>>>>> EAL: Detected CPU lcores: 40
>>>>> EAL: Detected NUMA nodes: 2
>>>>> EAL: Auto-detected process type: PRIMARY
>>>>> EAL: Detected static linkage of DPDK
>>>>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>>>>> EAL: Selected IOVA mode 'VA'
>>>>> EAL: VFIO support initialized
>>>>> EAL: Using IOMMU type 1 (Type 1)
>>>>> EAL: Probe PCI driver: net_nfp_pf (19ee:4000) device: 0000:81:00.0
>>>>> (socket 1) NFP HWINFO header: 48490200
>>>>> TELEMETRY: No legacy callbacks, legacy socket not created Set
>>>>> macswap packet forwarding mode
>>>>> testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=9600,
>>>>> socket=1
>>>>> testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port
>>>>> 0 (socket 1)
>>>>> Port0 dev_configure = -34
>>>>> Fail to configure port 0
>>>>> EAL: Error - exiting with code: 1
>>>>>      Cause: Start ports failed
>>>>>
>>>>> First  in the `nfp_net_configure()`, we will judge the value of MTU
>>>>> and hw- flbufsz, If MTU > hw->flbufsz, it will have the error.
>>>>>
>>>>> And the `--max-pkt-len` is setting the MTU in the initialize
>>>>> process, the
>>>> initialized  value of  hw->flbufsz is just 1500 at first.
>>>>>
>>>>> So if we set the `max-pkt-len`  bigger than the initialized value of
>>>>> flbufsz, It
>>>> will lead the error.
>>>>>
>>>>> Hence we set the new value of hw->flbufsz, it can large the range
>>>>> max-pkt-
>>>> len in the initialized process.
>>>>>
>>>>>
>>>>>>
>>>>>> This patch is fixing something but it is not clear what is fixed,
>>>>>> the concern is it may be changing driver to make something pass in
>>>>>> test
>>>> application (testpmd).
>>>>>>
>>>>>> What is 'flubfsz', is it Hw configured frame buffer size?
>>>>>
>>>>>
>>>>> It is configured in the `nfp_net_rx_queue_setup()`{`hw->flbufsz =
>>>>> rxq- mbuf_size`}.
>>>>> If the rxq->mbuf_size < MTU, the MTU can't work.
>>>>>
>>>>
>>>> It looks like `hw->flbufsz` holds the Rx buffer size, as you highlighted
>> above.
>>>>
>>>> And you don't want to accept frames bigger than buffer size, since it
>>>> seems driver doesn't support `RTE_ETH_RX_OFFLOAD_SCATTER`, all looks
>> OK.
>>>>
>>>>
>>>> According above logic, I agree "hw->flbufsz = RTE_ETHER_MTU;" is
>>>> wrong, but equally `hw->flbufsz = hw->max_mtu;` seems wrong.
>>>>
>>>> In above command line, it is safe because "mbuf-size=9600" and
>>>> "max-pkt-len=9216", buffer size is bigger than packet size.
>>> Yes, if I need set the hardcoded value, I should set the max max-pkt-len.
>>
>> I think it should be 'mbuf-size', since 'hw->flbufsz' is the buffer size.
>> 'max-pkt-len' is the max Rx frame accepted by NIC, for some devices 'max-
>> pkt-len' can be bigger than buffer size but it will work fine because of
>> segmented Rx, but it seems this config doesn't work for nfp.
>>
> As you said, maybe the ` NFP_FRAME_SIZE_MAX ` is right, for our nic,
> the max_rx_pktlen is NFP_FRAME_SIZE_MAX.

If 'NFP_FRAME_SIZE_MAX' is your HW limit, agree to set it if it will be 
a hardcoded value, please see below.

>>>>
>>>> You should able to set `hw->flbufsz` to current buffer size, instead
>>>> of a hardcoded value.
>>>>
>>>> In `nfp_net_init()`, most probably you don't know the buffer size
>>>> yet, can't you skip setting this value here and set it in
>>>> `nfp_net_rx_queue_setup()` when you know the buffer size?
>>>>
>>>
>>> But If I just depends on the `nfp_net_rx_queue_setup()`, in the
>>> `nfp_net_init()`, it will Call the  `nfp_net_configure()`, it will
>>> lead the testpmd start failed, so I add the hardcoded value in the initialize
>> process. Or  I can remove the judge about  `hw->flbufsz` in the
>> `nfp_net_init()`.
>>>
>>
>> Instead of setting 'hw->flbufsz' in 'nfp_net_init()' and check it in
>> 'nfp_net_configure()', why not set 'hw->flbufsz' in 'nfp_net_configure()'?
>>
>> Because in 'nfp_net_configure()', 'mtu' is a configuration parameter. It can
>> be possible to convert 'mtu' to frame size and set it. If there is a HW
>> limitation on buffer size, it can fail in 'nfp_net_configure()' when that limit hit.
>>
> You mean in the 'nfp_net_configure()', set the value of 'hw->flbufsz',
> Like
> `if (rxmode->mtu > hw->flbufsz) {
> 		hw->flbufsz = rxmode->mtu;
> 	}`
> Because the 'nfp_net_configure()' also isn't only called once.
> If it will be set the values every times, so I first just set the
> Initial value in the `nfp_net_inital()`.
> 


But during init, 'nfp_net_init()', you don't know the buffer size, that 
is why you are just setting a hardcoded value, which can be wrong, as 
you are observing now.

Also thinking twice setting "hw->flbufsz = rxmode->mtu" is not correct, 
since it is not frame size either.


What about:
- in 'nfp_net_configure()' change check as:
if (rxmode->mtu > NFP_FRAME_SIZE_MAX)
	return error

And you may want to save 'mtu' in driver or in device via
nn_cfg_writel(hw, NFP_NET_CFG_MTU, mtu);


- in start() add check for buffer size:
if (mtu > hw->flbufsz)
	return error

- in init() you can either remove setting 'hw->flbufsz' or set it to max 
(NFP_FRAME_SIZE_MAX) as you suggested.
If you remove setting it in init, you can have logic in configure() as
`
bufsize = hw->flbufsz ? hw->flbufsz : NFP_FRAME_SIZE_MAX;
if (rxmode->mtu > bufsize)
	return error
`

>>> Thanks for your advice.
>>>>
>>>>>>
>>>>>>
>>>>>>> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
>>>>>>> Cc: stable@dpdk.org
> ...
>>>>>>>          /* VLAN insertion is incompatible with LSOv2 */
>>>>>>>          if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
>>>>>
>>>
> 

RE: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Nole Zhang]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@amd.com>
> Sent: 2022年10月18日 17:36
> To: Nole Zhang <peng.zhang@corigine.com>; Chaoyong He
> <chaoyong.he@corigine.com>; dev@dpdk.org
> Cc: oss-drivers <oss-drivers@corigine.com>; Niklas Soderlund
> <niklas.soderlund@corigine.com>; stable@dpdk.org
> Subject: Re: [PATCH] net/nfp: set the appropriate initialized value of flbufsz
> 
> On 10/18/2022 9:51 AM, Nole Zhang wrote:
> >> On 10/18/2022 2:41 AM, Nole Zhang wrote:
> >>>
> >>>> On 10/15/2022 8:38 AM, Nole Zhang wrote:
> >>>>>     > On 10/10/2022 7:48 AM, Chaoyong He wrote:
> >>>>>>> From: Peng Zhang <peng.zhang@corigine.com>
> >>>>>>>
> >>>>>>> When the testpmd app start-up with parameter max-pkt-len, it
> >>>>>>> will set
> >>>>>> MTU.
> >>>>>>> But the initialized value of flubfsz is inappropriate, if the
> >>>>>>> value of flbufsz is smaller than the valude of max-pkt-len, the
> >>>>>>> testpmd app will start fail.
> >>>>>>>
> >>>>>>
> >>>>>> What is the failure in the testpmd?
> >>>>>
> >>>>> The log is as follows:
> >>>>> [root@volstruis ~]# dpdk-testpmd --main-lcore 10 -l 10,11,12 -n 4
> >>>>> -a
> >>>>> 0000:81:00.0 --socket-mem 2048,2048 --proc-type auto -- --portmask
> >>>>> 0x3 --nb-cores 2 --rxq 1 --txq 1 --rxd 1024 --txd 1024
> >>>>> --port-topology loop --forward-mode macswap  --max-pkt-len 9216
> >>>>> --mbuf-size 9600 --rss-udp --burst=32
> >>>>> EAL: Detected CPU lcores: 40
> >>>>> EAL: Detected NUMA nodes: 2
> >>>>> EAL: Auto-detected process type: PRIMARY
> >>>>> EAL: Detected static linkage of DPDK
> >>>>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >>>>> EAL: Selected IOVA mode 'VA'
> >>>>> EAL: VFIO support initialized
> >>>>> EAL: Using IOMMU type 1 (Type 1)
> >>>>> EAL: Probe PCI driver: net_nfp_pf (19ee:4000) device: 0000:81:00.0
> >>>>> (socket 1) NFP HWINFO header: 48490200
> >>>>> TELEMETRY: No legacy callbacks, legacy socket not created Set
> >>>>> macswap packet forwarding mode
> >>>>> testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=9600,
> >>>>> socket=1
> >>>>> testpmd: preferred mempool ops selected: ring_mp_mc Configuring
> >>>>> Port
> >>>>> 0 (socket 1)
> >>>>> Port0 dev_configure = -34
> >>>>> Fail to configure port 0
> >>>>> EAL: Error - exiting with code: 1
> >>>>>      Cause: Start ports failed
> >>>>>
> >>>>> First  in the `nfp_net_configure()`, we will judge the value of
> >>>>> MTU and hw- flbufsz, If MTU > hw->flbufsz, it will have the error.
> >>>>>
> >>>>> And the `--max-pkt-len` is setting the MTU in the initialize
> >>>>> process, the
> >>>> initialized  value of  hw->flbufsz is just 1500 at first.
> >>>>>
> >>>>> So if we set the `max-pkt-len`  bigger than the initialized value
> >>>>> of flbufsz, It
> >>>> will lead the error.
> >>>>>
> >>>>> Hence we set the new value of hw->flbufsz, it can large the range
> >>>>> max-pkt-
> >>>> len in the initialized process.
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> This patch is fixing something but it is not clear what is fixed,
> >>>>>> the concern is it may be changing driver to make something pass
> >>>>>> in test
> >>>> application (testpmd).
> >>>>>>
> >>>>>> What is 'flubfsz', is it Hw configured frame buffer size?
> >>>>>
> >>>>>
> >>>>> It is configured in the `nfp_net_rx_queue_setup()`{`hw->flbufsz =
> >>>>> rxq- mbuf_size`}.
> >>>>> If the rxq->mbuf_size < MTU, the MTU can't work.
> >>>>>
> >>>>
> >>>> It looks like `hw->flbufsz` holds the Rx buffer size, as you
> >>>> highlighted
> >> above.
> >>>>
> >>>> And you don't want to accept frames bigger than buffer size, since
> >>>> it seems driver doesn't support `RTE_ETH_RX_OFFLOAD_SCATTER`, all
> >>>> looks
> >> OK.
> >>>>
> >>>>
> >>>> According above logic, I agree "hw->flbufsz = RTE_ETHER_MTU;" is
> >>>> wrong, but equally `hw->flbufsz = hw->max_mtu;` seems wrong.
> >>>>
> >>>> In above command line, it is safe because "mbuf-size=9600" and
> >>>> "max-pkt-len=9216", buffer size is bigger than packet size.
> >>> Yes, if I need set the hardcoded value, I should set the max max-pkt-len.
> >>
> >> I think it should be 'mbuf-size', since 'hw->flbufsz' is the buffer size.
> >> 'max-pkt-len' is the max Rx frame accepted by NIC, for some devices
> >> 'max- pkt-len' can be bigger than buffer size but it will work fine
> >> because of segmented Rx, but it seems this config doesn't work for nfp.
> >>
> > As you said, maybe the ` NFP_FRAME_SIZE_MAX ` is right, for our nic,
> > the max_rx_pktlen is NFP_FRAME_SIZE_MAX.
> 
> If 'NFP_FRAME_SIZE_MAX' is your HW limit, agree to set it if it will be a
> hardcoded value, please see below.
> 
> >>>>
> >>>> You should able to set `hw->flbufsz` to current buffer size,
> >>>> instead of a hardcoded value.
> >>>>
> >>>> In `nfp_net_init()`, most probably you don't know the buffer size
> >>>> yet, can't you skip setting this value here and set it in
> >>>> `nfp_net_rx_queue_setup()` when you know the buffer size?
> >>>>
> >>>
> >>> But If I just depends on the `nfp_net_rx_queue_setup()`, in the
> >>> `nfp_net_init()`, it will Call the  `nfp_net_configure()`, it will
> >>> lead the testpmd start failed, so I add the hardcoded value in the
> >>> initialize
> >> process. Or  I can remove the judge about  `hw->flbufsz` in the
> >> `nfp_net_init()`.
> >>>
> >>
> >> Instead of setting 'hw->flbufsz' in 'nfp_net_init()' and check it in
> >> 'nfp_net_configure()', why not set 'hw->flbufsz' in 'nfp_net_configure()'?
> >>
> >> Because in 'nfp_net_configure()', 'mtu' is a configuration parameter.
> >> It can be possible to convert 'mtu' to frame size and set it. If
> >> there is a HW limitation on buffer size, it can fail in 'nfp_net_configure()'
> when that limit hit.
> >>
> > You mean in the 'nfp_net_configure()', set the value of 'hw->flbufsz',
> > Like `if (rxmode->mtu > hw->flbufsz) {
> > 		hw->flbufsz = rxmode->mtu;
> > 	}`
> > Because the 'nfp_net_configure()' also isn't only called once.
> > If it will be set the values every times, so I first just set the
> > Initial value in the `nfp_net_inital()`.
> >
> 
> 
> But during init, 'nfp_net_init()', you don't know the buffer size, that is why
> you are just setting a hardcoded value, which can be wrong, as you are
> observing now.
> 
> Also thinking twice setting "hw->flbufsz = rxmode->mtu" is not correct, since
> it is not frame size either.
> 
> 
> What about:
> - in 'nfp_net_configure()' change check as:
> if (rxmode->mtu > NFP_FRAME_SIZE_MAX)
> 	return error
> 
> And you may want to save 'mtu' in driver or in device via
> nn_cfg_writel(hw, NFP_NET_CFG_MTU, mtu);
> 
> 
> - in start() add check for buffer size:
> if (mtu > hw->flbufsz)
> 	return error
> 
> - in init() you can either remove setting 'hw->flbufsz' or set it to max
> (NFP_FRAME_SIZE_MAX) as you suggested.
> If you remove setting it in init, you can have logic in configure() as
> `
> bufsize = hw->flbufsz ? hw->flbufsz : NFP_FRAME_SIZE_MAX;
> if (rxmode->mtu > bufsize)
> 	return error
> `
Yes, thanks for your advice.

I will add the 
 - in start() add check for buffer size:
`if (mtu > hw->flbufsz)
	return error`
-in the init() keep the value
`
hw->flbufsz = NFP_FRAME_SIZE_MAX;
`
-in the conigdure() use this check
`
if (rxmode->mtu > NFP_FRAME_SIZE_MAX)
 	return error
`
I will send the v2 after the testing.

Thanks for your review and help again.

> 
> >>> Thanks for your advice.
> >>>>
> >>>>>>
> >>>>>>
> >>>>>>> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
> >>>>>>> Cc: stable@dpdk.org
> > ...
> >>>>>>>          /* VLAN insertion is incompatible with LSOv2 */
> >>>>>>>          if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
> >>>>>
> >>>
> >

Re: [PATCH] net/nfp: set the appropriate initialized value of flbufsz

[Ferruh Yigit]

On 10/18/2022 10:50 AM, Nole Zhang wrote:
> 
> 
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@amd.com>
>> Sent: 2022年10月18日 17:36
>> To: Nole Zhang <peng.zhang@corigine.com>; Chaoyong He
>> <chaoyong.he@corigine.com>; dev@dpdk.org
>> Cc: oss-drivers <oss-drivers@corigine.com>; Niklas Soderlund
>> <niklas.soderlund@corigine.com>; stable@dpdk.org
>> Subject: Re: [PATCH] net/nfp: set the appropriate initialized value of flbufsz
>>
>> On 10/18/2022 9:51 AM, Nole Zhang wrote:
>>>> On 10/18/2022 2:41 AM, Nole Zhang wrote:
>>>>>
>>>>>> On 10/15/2022 8:38 AM, Nole Zhang wrote:
>>>>>>>      > On 10/10/2022 7:48 AM, Chaoyong He wrote:
>>>>>>>>> From: Peng Zhang <peng.zhang@corigine.com>
>>>>>>>>>
>>>>>>>>> When the testpmd app start-up with parameter max-pkt-len, it
>>>>>>>>> will set
>>>>>>>> MTU.
>>>>>>>>> But the initialized value of flubfsz is inappropriate, if the
>>>>>>>>> value of flbufsz is smaller than the valude of max-pkt-len, the
>>>>>>>>> testpmd app will start fail.
>>>>>>>>>
>>>>>>>>
>>>>>>>> What is the failure in the testpmd?
>>>>>>>
>>>>>>> The log is as follows:
>>>>>>> [root@volstruis ~]# dpdk-testpmd --main-lcore 10 -l 10,11,12 -n 4
>>>>>>> -a
>>>>>>> 0000:81:00.0 --socket-mem 2048,2048 --proc-type auto -- --portmask
>>>>>>> 0x3 --nb-cores 2 --rxq 1 --txq 1 --rxd 1024 --txd 1024
>>>>>>> --port-topology loop --forward-mode macswap  --max-pkt-len 9216
>>>>>>> --mbuf-size 9600 --rss-udp --burst=32
>>>>>>> EAL: Detected CPU lcores: 40
>>>>>>> EAL: Detected NUMA nodes: 2
>>>>>>> EAL: Auto-detected process type: PRIMARY
>>>>>>> EAL: Detected static linkage of DPDK
>>>>>>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>>>>>>> EAL: Selected IOVA mode 'VA'
>>>>>>> EAL: VFIO support initialized
>>>>>>> EAL: Using IOMMU type 1 (Type 1)
>>>>>>> EAL: Probe PCI driver: net_nfp_pf (19ee:4000) device: 0000:81:00.0
>>>>>>> (socket 1) NFP HWINFO header: 48490200
>>>>>>> TELEMETRY: No legacy callbacks, legacy socket not created Set
>>>>>>> macswap packet forwarding mode
>>>>>>> testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=9600,
>>>>>>> socket=1
>>>>>>> testpmd: preferred mempool ops selected: ring_mp_mc Configuring
>>>>>>> Port
>>>>>>> 0 (socket 1)
>>>>>>> Port0 dev_configure = -34
>>>>>>> Fail to configure port 0
>>>>>>> EAL: Error - exiting with code: 1
>>>>>>>       Cause: Start ports failed
>>>>>>>
>>>>>>> First  in the `nfp_net_configure()`, we will judge the value of
>>>>>>> MTU and hw- flbufsz, If MTU > hw->flbufsz, it will have the error.
>>>>>>>
>>>>>>> And the `--max-pkt-len` is setting the MTU in the initialize
>>>>>>> process, the
>>>>>> initialized  value of  hw->flbufsz is just 1500 at first.
>>>>>>>
>>>>>>> So if we set the `max-pkt-len`  bigger than the initialized value
>>>>>>> of flbufsz, It
>>>>>> will lead the error.
>>>>>>>
>>>>>>> Hence we set the new value of hw->flbufsz, it can large the range
>>>>>>> max-pkt-
>>>>>> len in the initialized process.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> This patch is fixing something but it is not clear what is fixed,
>>>>>>>> the concern is it may be changing driver to make something pass
>>>>>>>> in test
>>>>>> application (testpmd).
>>>>>>>>
>>>>>>>> What is 'flubfsz', is it Hw configured frame buffer size?
>>>>>>>
>>>>>>>
>>>>>>> It is configured in the `nfp_net_rx_queue_setup()`{`hw->flbufsz =
>>>>>>> rxq- mbuf_size`}.
>>>>>>> If the rxq->mbuf_size < MTU, the MTU can't work.
>>>>>>>
>>>>>>
>>>>>> It looks like `hw->flbufsz` holds the Rx buffer size, as you
>>>>>> highlighted
>>>> above.
>>>>>>
>>>>>> And you don't want to accept frames bigger than buffer size, since
>>>>>> it seems driver doesn't support `RTE_ETH_RX_OFFLOAD_SCATTER`, all
>>>>>> looks
>>>> OK.
>>>>>>
>>>>>>
>>>>>> According above logic, I agree "hw->flbufsz = RTE_ETHER_MTU;" is
>>>>>> wrong, but equally `hw->flbufsz = hw->max_mtu;` seems wrong.
>>>>>>
>>>>>> In above command line, it is safe because "mbuf-size=9600" and
>>>>>> "max-pkt-len=9216", buffer size is bigger than packet size.
>>>>> Yes, if I need set the hardcoded value, I should set the max max-pkt-len.
>>>>
>>>> I think it should be 'mbuf-size', since 'hw->flbufsz' is the buffer size.
>>>> 'max-pkt-len' is the max Rx frame accepted by NIC, for some devices
>>>> 'max- pkt-len' can be bigger than buffer size but it will work fine
>>>> because of segmented Rx, but it seems this config doesn't work for nfp.
>>>>
>>> As you said, maybe the ` NFP_FRAME_SIZE_MAX ` is right, for our nic,
>>> the max_rx_pktlen is NFP_FRAME_SIZE_MAX.
>>
>> If 'NFP_FRAME_SIZE_MAX' is your HW limit, agree to set it if it will be a
>> hardcoded value, please see below.
>>
>>>>>>
>>>>>> You should able to set `hw->flbufsz` to current buffer size,
>>>>>> instead of a hardcoded value.
>>>>>>
>>>>>> In `nfp_net_init()`, most probably you don't know the buffer size
>>>>>> yet, can't you skip setting this value here and set it in
>>>>>> `nfp_net_rx_queue_setup()` when you know the buffer size?
>>>>>>
>>>>>
>>>>> But If I just depends on the `nfp_net_rx_queue_setup()`, in the
>>>>> `nfp_net_init()`, it will Call the  `nfp_net_configure()`, it will
>>>>> lead the testpmd start failed, so I add the hardcoded value in the
>>>>> initialize
>>>> process. Or  I can remove the judge about  `hw->flbufsz` in the
>>>> `nfp_net_init()`.
>>>>>
>>>>
>>>> Instead of setting 'hw->flbufsz' in 'nfp_net_init()' and check it in
>>>> 'nfp_net_configure()', why not set 'hw->flbufsz' in 'nfp_net_configure()'?
>>>>
>>>> Because in 'nfp_net_configure()', 'mtu' is a configuration parameter.
>>>> It can be possible to convert 'mtu' to frame size and set it. If
>>>> there is a HW limitation on buffer size, it can fail in 'nfp_net_configure()'
>> when that limit hit.
>>>>
>>> You mean in the 'nfp_net_configure()', set the value of 'hw->flbufsz',
>>> Like `if (rxmode->mtu > hw->flbufsz) {
>>> 		hw->flbufsz = rxmode->mtu;
>>> 	}`
>>> Because the 'nfp_net_configure()' also isn't only called once.
>>> If it will be set the values every times, so I first just set the
>>> Initial value in the `nfp_net_inital()`.
>>>
>>
>>
>> But during init, 'nfp_net_init()', you don't know the buffer size, that is why
>> you are just setting a hardcoded value, which can be wrong, as you are
>> observing now.
>>
>> Also thinking twice setting "hw->flbufsz = rxmode->mtu" is not correct, since
>> it is not frame size either.
>>
>>
>> What about:
>> - in 'nfp_net_configure()' change check as:
>> if (rxmode->mtu > NFP_FRAME_SIZE_MAX)
>> 	return error
>>
>> And you may want to save 'mtu' in driver or in device via
>> nn_cfg_writel(hw, NFP_NET_CFG_MTU, mtu);
>>
>>
>> - in start() add check for buffer size:
>> if (mtu > hw->flbufsz)
>> 	return error
>>
>> - in init() you can either remove setting 'hw->flbufsz' or set it to max
>> (NFP_FRAME_SIZE_MAX) as you suggested.
>> If you remove setting it in init, you can have logic in configure() as
>> `
>> bufsize = hw->flbufsz ? hw->flbufsz : NFP_FRAME_SIZE_MAX;
>> if (rxmode->mtu > bufsize)
>> 	return error
>> `
> Yes, thanks for your advice.
> 
> I will add the
>   - in start() add check for buffer size:
> `if (mtu > hw->flbufsz)
> 	return error`
> -in the init() keep the value
> `
> hw->flbufsz = NFP_FRAME_SIZE_MAX;
> `
> -in the conigdure() use this check
> `
> if (rxmode->mtu > NFP_FRAME_SIZE_MAX)
>   	return error
> `

If you will initialize 'hw->flbufsz' as 'NFP_FRAME_SIZE_MAX' in the 
init(), you can keep the check in the configure() intact, up to you.

> I will send the v2 after the testing.
> 

ack, thanks.

> Thanks for your review and help again.
> 
>>
>>>>> Thanks for your advice.
>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
>>>>>>>>> Cc: stable@dpdk.org
>>> ...
>>>>>>>>>           /* VLAN insertion is incompatible with LSOv2 */
>>>>>>>>>           if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
>>>>>>>
>>>>>
>>>
> 

[PATCH v2] net/nfp: ensure the MTU can work

[Chaoyong He]

From: Peng Zhang <peng.zhang@corigine.com>

When MTU is bigger than hw->flbufsz, it can't work. hw->flbufsz is set in
the nfp_net_rx_queue_setup().

At first, in the nfp_net_configure(), the hw->flbufsz isn't set the value,
it just judge the initialized value and MTU, it is unreasonable.

Now, it just check the MTU can't be more than the NFP_FRAME_SIZE_MAX in
the nfp_net_configure(), when hw->flbufsz is set the value, in the
nfp_net_start(), judge the hw->flbufsz and MTU.

Fixes: 5c305e218f15 ("net/nfp: fix initialization")
Cc: stable@dpdk.org

Signed-off-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>

* Changes since v1
- Remove the assignment in init() and move the check to nfp_net_start().

---
 drivers/net/nfp/nfp_common.c    | 6 +++---
 drivers/net/nfp/nfp_ethdev.c    | 8 +++++++-
 drivers/net/nfp/nfp_ethdev_vf.c | 1 -
 3 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/net/nfp/nfp_common.c b/drivers/net/nfp/nfp_common.c
index 0e55f0c..289c70f 100644
--- a/drivers/net/nfp/nfp_common.c
+++ b/drivers/net/nfp/nfp_common.c
@@ -177,9 +177,9 @@
 	}
 
 	/* Checking MTU set */
-	if (rxmode->mtu > hw->flbufsz) {
-		PMD_INIT_LOG(INFO, "MTU (%u) larger then current mbufsize (%u) not supported",
-				    rxmode->mtu, hw->flbufsz);
+	if (rxmode->mtu > NFP_FRAME_SIZE_MAX) {
+		PMD_INIT_LOG(ERR, "MTU (%u) larger than NFP_FRAME_SIZE_MAX (%u) not supported",
+				    rxmode->mtu, NFP_FRAME_SIZE_MAX);
 		return -ERANGE;
 	}
 
diff --git a/drivers/net/nfp/nfp_ethdev.c b/drivers/net/nfp/nfp_ethdev.c
index 5cdd34e..3b952fa 100644
--- a/drivers/net/nfp/nfp_ethdev.c
+++ b/drivers/net/nfp/nfp_ethdev.c
@@ -110,6 +110,13 @@
 		update = NFP_NET_CFG_UPDATE_MSIX;
 	}
 
+	/* Checking MTU set */
+	if (dev->data->mtu > hw->flbufsz) {
+		PMD_INIT_LOG(ERR, "MTU (%u) can't be larger than the current NFP_FRAME_SIZE (%u)",
+				dev->data->mtu, hw->flbufsz);
+		return -ERANGE;
+	}
+
 	rte_intr_enable(intr_handle);
 
 	new_ctrl = nfp_check_offloads(dev);
@@ -517,7 +524,6 @@
 	hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
 	hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
 	hw->mtu = RTE_ETHER_MTU;
-	hw->flbufsz = RTE_ETHER_MTU;
 
 	/* VLAN insertion is incompatible with LSOv2 */
 	if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
diff --git a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
index d304d78..3f4ad3e 100644
--- a/drivers/net/nfp/nfp_ethdev_vf.c
+++ b/drivers/net/nfp/nfp_ethdev_vf.c
@@ -396,7 +396,6 @@
 	hw->cap = nn_cfg_readl(hw, NFP_NET_CFG_CAP);
 	hw->max_mtu = nn_cfg_readl(hw, NFP_NET_CFG_MAX_MTU);
 	hw->mtu = RTE_ETHER_MTU;
-	hw->flbufsz = RTE_ETHER_MTU;
 
 	/* VLAN insertion is incompatible with LSOv2 */
 	if (hw->cap & NFP_NET_CFG_CTRL_LSO2)
-- 
1.8.3.1

Re: [PATCH v2] net/nfp: ensure the MTU can work

[Ferruh Yigit]

On 10/21/2022 7:27 AM, Chaoyong He wrote:
> From: Peng Zhang <peng.zhang@corigine.com>
> 
> When MTU is bigger than hw->flbufsz, it can't work. hw->flbufsz is set in
> the nfp_net_rx_queue_setup().
> 
> At first, in the nfp_net_configure(), the hw->flbufsz isn't set the value,
> it just judge the initialized value and MTU, it is unreasonable.
> 
> Now, it just check the MTU can't be more than the NFP_FRAME_SIZE_MAX in
> the nfp_net_configure(), when hw->flbufsz is set the value, in the
> nfp_net_start(), judge the hw->flbufsz and MTU.
> 
> Fixes: 5c305e218f15 ("net/nfp: fix initialization")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Peng Zhang <peng.zhang@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
> 

Applied to dpdk-next-net/main, thanks.