[dpdk-users] MLX5: Using packet send scheduling / packet pacing

[dpdk-users] MLX5: Using packet send scheduling / packet pacing

[Carsten Andrich]

Hello everyone,

I'm trying to use packet send scheduling [1] with DPDK 20.11 and the 
MLX5 PMD (NIC: ConnectX-5 MCX516A-CDAT). This patch contains some 
additional information on this feature also know as packet pacing [2].

According to MLX5's docs, packet pacing requires the "tx_pp" parameter 
[3, CTRL+F: "tx_pp"]. However, when firing up testpmd with that 
parameter, it fails as follows:

> # dpdk-testpmd -a 81:00.0,tx_pp=500 -- -i
> ...
> EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:81:00.0 (socket 0)
> mlx5_pci: WQE rate mode is required for packet pacing
> mlx5_pci: probe of PCI device 0000:81:00.0 aborted after encountering an error: No such device
> common_mlx5: Failed to load driver = mlx5_pci.
>
> EAL: Requested device 0000:81:00.0 cannot be used
The error message originates here [4] and is caused by what to me 
appears to be a value read from the NIC [5]. Unfortunately, that leaves 
me clueless on how to activate the required "WQE rate mode".  According 
to the output of ibv_devinfo, my NIC does support packet pacing:

> # ibv_devinfo -v 81:00.0
> ...
> 	packet_pacing_caps:
> 		qp_rate_limit_min:	1kbps
> 		qp_rate_limit_max:	100000000kbps
> 		supported_qp:
> 					SUPPORT_RAW_PACKET
I'd be grateful for any information on how to get packet pacing up and 
running. Am I just missing another required option (which is not given 
in the docs) or does my NIC lack packet pacing support?

Thank you very much in advance.

Best regards,
Carsten

[1] 
https://doc.dpdk.org/api/rte__ethdev_8h.html#a990d8351447a710628cbb24a28d3252d
[2] https://patches.dpdk.org/patch/73742/
[3] https://doc.dpdk.org/guides/nics/mlx5.html#run-time-configuration
[4] 
http://code.dpdk.org/dpdk/v20.11/source/drivers/net/mlx5/linux/mlx5_os.c#L1278
[5] 
http://code.dpdk.org/dpdk/v20.11/source/drivers/common/mlx5/mlx5_devx_cmds.c#L748

Re: [dpdk-users] MLX5: Using packet send scheduling / packet pacing

[Thomas Monjalon]

+Cc Slava

29/01/2021 17:30, Carsten Andrich:
> Hello everyone,
> 
> I'm trying to use packet send scheduling [1] with DPDK 20.11 and the 
> MLX5 PMD (NIC: ConnectX-5 MCX516A-CDAT). This patch contains some 
> additional information on this feature also know as packet pacing [2].
> 
> According to MLX5's docs, packet pacing requires the "tx_pp" parameter 
> [3, CTRL+F: "tx_pp"]. However, when firing up testpmd with that 
> parameter, it fails as follows:
> 
> > # dpdk-testpmd -a 81:00.0,tx_pp=500 -- -i
> > ...
> > EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:81:00.0 (socket 0)
> > mlx5_pci: WQE rate mode is required for packet pacing
> > mlx5_pci: probe of PCI device 0000:81:00.0 aborted after encountering an error: No such device
> > common_mlx5: Failed to load driver = mlx5_pci.
> >
> > EAL: Requested device 0000:81:00.0 cannot be used
> The error message originates here [4] and is caused by what to me 
> appears to be a value read from the NIC [5]. Unfortunately, that leaves 
> me clueless on how to activate the required "WQE rate mode".  According 
> to the output of ibv_devinfo, my NIC does support packet pacing:
> 
> > # ibv_devinfo -v 81:00.0
> > ...
> > 	packet_pacing_caps:
> > 		qp_rate_limit_min:	1kbps
> > 		qp_rate_limit_max:	100000000kbps
> > 		supported_qp:
> > 					SUPPORT_RAW_PACKET
> I'd be grateful for any information on how to get packet pacing up and 
> running. Am I just missing another required option (which is not given 
> in the docs) or does my NIC lack packet pacing support?
> 
> Thank you very much in advance.
> 
> Best regards,
> Carsten
> 
> [1] 
> https://doc.dpdk.org/api/rte__ethdev_8h.html#a990d8351447a710628cbb24a28d3252d
> [2] https://patches.dpdk.org/patch/73742/
> [3] https://doc.dpdk.org/guides/nics/mlx5.html#run-time-configuration
> [4] 
> http://code.dpdk.org/dpdk/v20.11/source/drivers/net/mlx5/linux/mlx5_os.c#L1278
> [5] 
> http://code.dpdk.org/dpdk/v20.11/source/drivers/common/mlx5/mlx5_devx_cmds.c#L748
> 
> 

Re: [dpdk-users] MLX5: Using packet send scheduling / packet pacing

[Slava Ovsiienko]

Hi, Carsten

To use packet pacing capability:
- ConnectX-6DX or above is required
- the NV settings should be configured (set timestamp to real-time UTC)

With best regards,
Slava

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Friday, January 29, 2021 18:50
> To: Carsten Andrich <carsten.andrich@tu-ilmenau.de>
> Cc: users@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com>
> Subject: Re: [dpdk-users] MLX5: Using packet send scheduling / packet pacing
> 
> 29/01/2021 17:30, Carsten Andrich:
> > Hello everyone,
> >
> > I'm trying to use packet send scheduling [1] with DPDK 20.11 and the
> > MLX5 PMD (NIC: ConnectX-5 MCX516A-CDAT). This patch contains some
> > additional information on this feature also know as packet pacing [2].
> >
> > According to MLX5's docs, packet pacing requires the "tx_pp" parameter
> > [3, CTRL+F: "tx_pp"]. However, when firing up testpmd with that
> > parameter, it fails as follows:
> >
> > > # dpdk-testpmd -a 81:00.0,tx_pp=500 -- -i ...
> > > EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:81:00.0
> > > (socket 0)
> > > mlx5_pci: WQE rate mode is required for packet pacing
> > > mlx5_pci: probe of PCI device 0000:81:00.0 aborted after
> > > encountering an error: No such device
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:81:00.0 cannot be used
> > The error message originates here [4] and is caused by what to me
> > appears to be a value read from the NIC [5]. Unfortunately, that
> > leaves me clueless on how to activate the required "WQE rate mode".
> > According to the output of ibv_devinfo, my NIC does support packet pacing:
> >
> > > # ibv_devinfo -v 81:00.0
> > > ...
> > > 	packet_pacing_caps:
> > > 		qp_rate_limit_min:	1kbps
> > > 		qp_rate_limit_max:	100000000kbps
> > > 		supported_qp:
> > > 					SUPPORT_RAW_PACKET
> > I'd be grateful for any information on how to get packet pacing up and
> > running. Am I just missing another required option (which is not given
> > in the docs) or does my NIC lack packet pacing support?
> >
> > Thank you very much in advance.
> >
> > Best regards,
> > Carsten
> >
> > [1]
> >
> https://doc.dpdk.org/api/rte__ethdev_8h.html#a990d8351447a710628cbb24
> a
> > 28d3252d [2] https://patches.dpdk.org/patch/73742/
> > [3] https://doc.dpdk.org/guides/nics/mlx5.html#run-time-configuration
> > [4]
> > http://code.dpdk.org/dpdk/v20.11/source/drivers/net/mlx5/linux/mlx5_os
> > .c#L1278
> > [5]
> >
> http://code.dpdk.org/dpdk/v20.11/source/drivers/common/mlx5/mlx5_devx
> _
> > cmds.c#L748
> >
> >
> 
> 
> 
> 

Re: [dpdk-users] MLX5: Using packet send scheduling / packet pacing

[Carsten Andrich]

Hi Slava,

thank you for the prompt response. I think the requirements for Packet 
Pacing should be added to Table 34.2(?) of the MLX5 docs [1].

Best regards,
Carsten

[1] https://doc.dpdk.org/guides/nics/mlx5.html#mlx5-offloads-support

On 29.01.21 19:04, Slava Ovsiienko wrote:
> Hi, Carsten
>
> To use packet pacing capability:
> - ConnectX-6DX or above is required
> - the NV settings should be configured (set timestamp to real-time UTC)
>
> With best regards,
> Slava
>
>> -----Original Message-----
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Friday, January 29, 2021 18:50
>> To: Carsten Andrich <carsten.andrich@tu-ilmenau.de>
>> Cc: users@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com>
>> Subject: Re: [dpdk-users] MLX5: Using packet send scheduling / packet pacing
>>
>> 29/01/2021 17:30, Carsten Andrich:
>>> Hello everyone,
>>>
>>> I'm trying to use packet send scheduling [1] with DPDK 20.11 and the
>>> MLX5 PMD (NIC: ConnectX-5 MCX516A-CDAT). This patch contains some
>>> additional information on this feature also know as packet pacing [2].
>>>
>>> According to MLX5's docs, packet pacing requires the "tx_pp" parameter
>>> [3, CTRL+F: "tx_pp"]. However, when firing up testpmd with that
>>> parameter, it fails as follows:
>>>
>>>> # dpdk-testpmd -a 81:00.0,tx_pp=500 -- -i ...
>>>> EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:81:00.0
>>>> (socket 0)
>>>> mlx5_pci: WQE rate mode is required for packet pacing
>>>> mlx5_pci: probe of PCI device 0000:81:00.0 aborted after
>>>> encountering an error: No such device
>>>> common_mlx5: Failed to load driver = mlx5_pci.
>>>>
>>>> EAL: Requested device 0000:81:00.0 cannot be used
>>> The error message originates here [4] and is caused by what to me
>>> appears to be a value read from the NIC [5]. Unfortunately, that
>>> leaves me clueless on how to activate the required "WQE rate mode".
>>> According to the output of ibv_devinfo, my NIC does support packet pacing:
>>>
>>>> # ibv_devinfo -v 81:00.0
>>>> ...
>>>> 	packet_pacing_caps:
>>>> 		qp_rate_limit_min:	1kbps
>>>> 		qp_rate_limit_max:	100000000kbps
>>>> 		supported_qp:
>>>> 					SUPPORT_RAW_PACKET
>>> I'd be grateful for any information on how to get packet pacing up and
>>> running. Am I just missing another required option (which is not given
>>> in the docs) or does my NIC lack packet pacing support?
>>>
>>> Thank you very much in advance.
>>>
>>> Best regards,
>>> Carsten
>>>
>>> [1]
>>>
>> https://doc.dpdk.org/api/rte__ethdev_8h.html#a990d8351447a710628cbb24
>> a
>>> 28d3252d [2] https://patches.dpdk.org/patch/73742/
>>> [3] https://doc.dpdk.org/guides/nics/mlx5.html#run-time-configuration
>>> [4]
>>> http://code.dpdk.org/dpdk/v20.11/source/drivers/net/mlx5/linux/mlx5_os
>>> .c#L1278
>>> [5]
>>>
>> http://code.dpdk.org/dpdk/v20.11/source/drivers/common/mlx5/mlx5_devx
>> _
>>> cmds.c#L748
>>>
>>>
>>
>>
>>

Re: [dpdk-users] MLX5: Using packet send scheduling / packet pacing

[Thomas Monjalon]

30/01/2021 11:54, Carsten Andrich:
> Hi Slava,
> 
> thank you for the prompt response. I think the requirements for Packet 
> Pacing should be added to Table 34.2(?) of the MLX5 docs [1].

+1 for improving the doc