Hello everyone, I'm trying to use packet send scheduling [1] with DPDK 20.11 and the MLX5 PMD (NIC: ConnectX-5 MCX516A-CDAT). This patch contains some additional information on this feature also know as packet pacing [2]. According to MLX5's docs, packet pacing requires the "tx_pp" parameter [3, CTRL+F: "tx_pp"]. However, when firing up testpmd with that parameter, it fails as follows: > # dpdk-testpmd -a 81:00.0,tx_pp=500 -- -i > ... > EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:81:00.0 (socket 0) > mlx5_pci: WQE rate mode is required for packet pacing > mlx5_pci: probe of PCI device 0000:81:00.0 aborted after encountering an error: No such device > common_mlx5: Failed to load driver = mlx5_pci. > > EAL: Requested device 0000:81:00.0 cannot be used The error message originates here [4] and is caused by what to me appears to be a value read from the NIC [5]. Unfortunately, that leaves me clueless on how to activate the required "WQE rate mode". According to the output of ibv_devinfo, my NIC does support packet pacing: > # ibv_devinfo -v 81:00.0 > ... > packet_pacing_caps: > qp_rate_limit_min: 1kbps > qp_rate_limit_max: 100000000kbps > supported_qp: > SUPPORT_RAW_PACKET I'd be grateful for any information on how to get packet pacing up and running. Am I just missing another required option (which is not given in the docs) or does my NIC lack packet pacing support? Thank you very much in advance. Best regards, Carsten [1] https://doc.dpdk.org/api/rte__ethdev_8h.html#a990d8351447a710628cbb24a28d3252d [2] https://patches.dpdk.org/patch/73742/ [3] https://doc.dpdk.org/guides/nics/mlx5.html#run-time-configuration [4] http://code.dpdk.org/dpdk/v20.11/source/drivers/net/mlx5/linux/mlx5_os.c#L1278 [5] http://code.dpdk.org/dpdk/v20.11/source/drivers/common/mlx5/mlx5_devx_cmds.c#L748
+Cc Slava
29/01/2021 17:30, Carsten Andrich:
> Hello everyone,
>
> I'm trying to use packet send scheduling [1] with DPDK 20.11 and the
> MLX5 PMD (NIC: ConnectX-5 MCX516A-CDAT). This patch contains some
> additional information on this feature also know as packet pacing [2].
>
> According to MLX5's docs, packet pacing requires the "tx_pp" parameter
> [3, CTRL+F: "tx_pp"]. However, when firing up testpmd with that
> parameter, it fails as follows:
>
> > # dpdk-testpmd -a 81:00.0,tx_pp=500 -- -i
> > ...
> > EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:81:00.0 (socket 0)
> > mlx5_pci: WQE rate mode is required for packet pacing
> > mlx5_pci: probe of PCI device 0000:81:00.0 aborted after encountering an error: No such device
> > common_mlx5: Failed to load driver = mlx5_pci.
> >
> > EAL: Requested device 0000:81:00.0 cannot be used
> The error message originates here [4] and is caused by what to me
> appears to be a value read from the NIC [5]. Unfortunately, that leaves
> me clueless on how to activate the required "WQE rate mode". According
> to the output of ibv_devinfo, my NIC does support packet pacing:
>
> > # ibv_devinfo -v 81:00.0
> > ...
> > packet_pacing_caps:
> > qp_rate_limit_min: 1kbps
> > qp_rate_limit_max: 100000000kbps
> > supported_qp:
> > SUPPORT_RAW_PACKET
> I'd be grateful for any information on how to get packet pacing up and
> running. Am I just missing another required option (which is not given
> in the docs) or does my NIC lack packet pacing support?
>
> Thank you very much in advance.
>
> Best regards,
> Carsten
>
> [1]
> https://doc.dpdk.org/api/rte__ethdev_8h.html#a990d8351447a710628cbb24a28d3252d
> [2] https://patches.dpdk.org/patch/73742/
> [3] https://doc.dpdk.org/guides/nics/mlx5.html#run-time-configuration
> [4]
> http://code.dpdk.org/dpdk/v20.11/source/drivers/net/mlx5/linux/mlx5_os.c#L1278
> [5]
> http://code.dpdk.org/dpdk/v20.11/source/drivers/common/mlx5/mlx5_devx_cmds.c#L748
>
>
Hi, Carsten
To use packet pacing capability:
- ConnectX-6DX or above is required
- the NV settings should be configured (set timestamp to real-time UTC)
With best regards,
Slava
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Friday, January 29, 2021 18:50
> To: Carsten Andrich <carsten.andrich@tu-ilmenau.de>
> Cc: users@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com>
> Subject: Re: [dpdk-users] MLX5: Using packet send scheduling / packet pacing
>
> 29/01/2021 17:30, Carsten Andrich:
> > Hello everyone,
> >
> > I'm trying to use packet send scheduling [1] with DPDK 20.11 and the
> > MLX5 PMD (NIC: ConnectX-5 MCX516A-CDAT). This patch contains some
> > additional information on this feature also know as packet pacing [2].
> >
> > According to MLX5's docs, packet pacing requires the "tx_pp" parameter
> > [3, CTRL+F: "tx_pp"]. However, when firing up testpmd with that
> > parameter, it fails as follows:
> >
> > > # dpdk-testpmd -a 81:00.0,tx_pp=500 -- -i ...
> > > EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:81:00.0
> > > (socket 0)
> > > mlx5_pci: WQE rate mode is required for packet pacing
> > > mlx5_pci: probe of PCI device 0000:81:00.0 aborted after
> > > encountering an error: No such device
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:81:00.0 cannot be used
> > The error message originates here [4] and is caused by what to me
> > appears to be a value read from the NIC [5]. Unfortunately, that
> > leaves me clueless on how to activate the required "WQE rate mode".
> > According to the output of ibv_devinfo, my NIC does support packet pacing:
> >
> > > # ibv_devinfo -v 81:00.0
> > > ...
> > > packet_pacing_caps:
> > > qp_rate_limit_min: 1kbps
> > > qp_rate_limit_max: 100000000kbps
> > > supported_qp:
> > > SUPPORT_RAW_PACKET
> > I'd be grateful for any information on how to get packet pacing up and
> > running. Am I just missing another required option (which is not given
> > in the docs) or does my NIC lack packet pacing support?
> >
> > Thank you very much in advance.
> >
> > Best regards,
> > Carsten
> >
> > [1]
> >
> https://doc.dpdk.org/api/rte__ethdev_8h.html#a990d8351447a710628cbb24
> a
> > 28d3252d [2] https://patches.dpdk.org/patch/73742/
> > [3] https://doc.dpdk.org/guides/nics/mlx5.html#run-time-configuration
> > [4]
> > http://code.dpdk.org/dpdk/v20.11/source/drivers/net/mlx5/linux/mlx5_os
> > .c#L1278
> > [5]
> >
> http://code.dpdk.org/dpdk/v20.11/source/drivers/common/mlx5/mlx5_devx
> _
> > cmds.c#L748
> >
> >
>
>
>
>
Hi Slava, thank you for the prompt response. I think the requirements for Packet Pacing should be added to Table 34.2(?) of the MLX5 docs [1]. Best regards, Carsten [1] https://doc.dpdk.org/guides/nics/mlx5.html#mlx5-offloads-support On 29.01.21 19:04, Slava Ovsiienko wrote: > Hi, Carsten > > To use packet pacing capability: > - ConnectX-6DX or above is required > - the NV settings should be configured (set timestamp to real-time UTC) > > With best regards, > Slava > >> -----Original Message----- >> From: Thomas Monjalon <thomas@monjalon.net> >> Sent: Friday, January 29, 2021 18:50 >> To: Carsten Andrich <carsten.andrich@tu-ilmenau.de> >> Cc: users@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com> >> Subject: Re: [dpdk-users] MLX5: Using packet send scheduling / packet pacing >> >> 29/01/2021 17:30, Carsten Andrich: >>> Hello everyone, >>> >>> I'm trying to use packet send scheduling [1] with DPDK 20.11 and the >>> MLX5 PMD (NIC: ConnectX-5 MCX516A-CDAT). This patch contains some >>> additional information on this feature also know as packet pacing [2]. >>> >>> According to MLX5's docs, packet pacing requires the "tx_pp" parameter >>> [3, CTRL+F: "tx_pp"]. However, when firing up testpmd with that >>> parameter, it fails as follows: >>> >>>> # dpdk-testpmd -a 81:00.0,tx_pp=500 -- -i ... >>>> EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:81:00.0 >>>> (socket 0) >>>> mlx5_pci: WQE rate mode is required for packet pacing >>>> mlx5_pci: probe of PCI device 0000:81:00.0 aborted after >>>> encountering an error: No such device >>>> common_mlx5: Failed to load driver = mlx5_pci. >>>> >>>> EAL: Requested device 0000:81:00.0 cannot be used >>> The error message originates here [4] and is caused by what to me >>> appears to be a value read from the NIC [5]. Unfortunately, that >>> leaves me clueless on how to activate the required "WQE rate mode". >>> According to the output of ibv_devinfo, my NIC does support packet pacing: >>> >>>> # ibv_devinfo -v 81:00.0 >>>> ... >>>> packet_pacing_caps: >>>> qp_rate_limit_min: 1kbps >>>> qp_rate_limit_max: 100000000kbps >>>> supported_qp: >>>> SUPPORT_RAW_PACKET >>> I'd be grateful for any information on how to get packet pacing up and >>> running. Am I just missing another required option (which is not given >>> in the docs) or does my NIC lack packet pacing support? >>> >>> Thank you very much in advance. >>> >>> Best regards, >>> Carsten >>> >>> [1] >>> >> https://doc.dpdk.org/api/rte__ethdev_8h.html#a990d8351447a710628cbb24 >> a >>> 28d3252d [2] https://patches.dpdk.org/patch/73742/ >>> [3] https://doc.dpdk.org/guides/nics/mlx5.html#run-time-configuration >>> [4] >>> http://code.dpdk.org/dpdk/v20.11/source/drivers/net/mlx5/linux/mlx5_os >>> .c#L1278 >>> [5] >>> >> http://code.dpdk.org/dpdk/v20.11/source/drivers/common/mlx5/mlx5_devx >> _ >>> cmds.c#L748 >>> >>> >> >> >>
30/01/2021 11:54, Carsten Andrich:
> Hi Slava,
>
> thank you for the prompt response. I think the requirements for Packet
> Pacing should be added to Table 34.2(?) of the MLX5 docs [1].
+1 for improving the doc