[mlx5 driver] Usage of mlx5 with unpriviledged LXC container

[mlx5 driver] Usage of mlx5 with unpriviledged LXC container

[Julien]

[-- Attachment #1: Type: text/plain, Size: 4046 bytes --]

Hello,
I have a question about using the mlx5 driver with LXC.
I'm trying to use dpdk-testpmd in an LXC container whose root user isn't
mapped to the host's root user.
Note: The entire physical interface is given to the LXC container, not a
virtual interface.

The following error occured:
mlx5_common: DevX create TIS failed errno=22 status=0 syndrome=0
mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_2.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:27:00.0 aborted after encountering an
error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:27:00.0 cannot be used
EAL: Bus (pci) probe failed.

The "transport_domain" is created, and the mlx5_devx_cmd_create_td()
function runs normally.
The call to mlx5dv_devx_obj_create() receives an errno of 22.

I don't encounter any problems when the container's root user is mapped to
the host's root user.
Has anyone experienced this before?
Is it possible to use the driver in an unprivileged LXC container?

Dpdk version: 23.11
Linux Kernel: 5.15
Stack:
libibverbs.so.1!execute_ioctl(struct ibv_context * context, struct
ibv_context * context@entry, struct ibv_command_buffer * cmd, struct
ibv_command_buffer * cmd@entry) (\rdma-core-49.0\libibverbs\cmd_ioctl.c:147)
libmlx5.so.1!_mlx5dv_devx_obj_create(struct ibv_context * context, const
void * in, size_t inlen, void * out, size_t outlen)
(\rdma-core-49.0\providers\mlx5\verbs.c:5794)
libmlx5.so.1!mlx5dv_devx_obj_create(struct ibv_context * context, const
void * in, size_t inlen, void * out, size_t outlen)
(\rdma-core-49.0\providers\mlx5\verbs.c:5819)
librte_common_mlx5.so.24!mlx5_glue_devx_obj_create(struct ibv_context *
ctx, const void * in, size_t inlen, void * out, size_t outlen)
(\dpdk-23.11\drivers\common\mlx5\linux\mlx5_glue.c:1045)
librte_common_mlx5.so.24!mlx5_devx_cmd_create_tis(void * ctx, struct
mlx5_devx_tis_attr * tis_attr)
(\dpdk-23.11\drivers\common\mlx5\mlx5_devx_cmds.c:2037)
librte_net_mlx5.so!mlx5_setup_tis(struct mlx5_dev_ctx_shared * sh)
(\dpdk-23.11\drivers\net\mlx5\mlx5.c:1343)
librte_net_mlx5.so!mlx5_alloc_shared_dev_ctx(const struct
mlx5_dev_spawn_data * spawn, struct mlx5_kvargs_ctrl * mkvlist)
(\dpdk-23.11\drivers\net\mlx5\mlx5.c:1784)
librte_net_mlx5.so!mlx5_dev_spawn(struct rte_device * dpdk_dev, struct
mlx5_dev_spawn_data * spawn, struct rte_eth_devargs * eth_da, struct
mlx5_kvargs_ctrl * mkvlist)
(\dpdk-23.11\drivers\net\mlx5\linux\mlx5_os.c:1169)
librte_net_mlx5.so!mlx5_os_pci_probe_pf(struct mlx5_common_device * cdev,
struct rte_eth_devargs * req_eth_da, uint16_t owner_id, struct
mlx5_kvargs_ctrl * mkvlist)
(\dpdk-23.11\drivers\net\mlx5\linux\mlx5_os.c:2648)
librte_net_mlx5.so!mlx5_os_pci_probe(struct mlx5_common_device * cdev,
struct mlx5_kvargs_ctrl * mkvlist)
(\dpdk-23.11\drivers\net\mlx5\linux\mlx5_os.c:2797)
librte_net_mlx5.so!mlx5_os_net_probe(struct mlx5_common_device * cdev,
struct mlx5_kvargs_ctrl * mkvlist)
(\dpdk-23.11\drivers\net\mlx5\linux\mlx5_os.c:2881)
librte_common_mlx5.so.24!drivers_probe(struct mlx5_common_device * cdev,
uint32_t user_classes, struct mlx5_kvargs_ctrl * mkvlist)
(\dpdk-23.11\drivers\common\mlx5\mlx5_common.c:938)
librte_common_mlx5.so.24!mlx5_common_dev_probe(struct rte_device * eal_dev)
(\dpdk-23.11\drivers\common\mlx5\mlx5_common.c:1028)
librte_common_mlx5.so.24!mlx5_common_pci_probe(struct rte_pci_driver *
pci_drv, struct rte_pci_device * pci_dev)
(\dpdk-23.11\drivers\common\mlx5\mlx5_common_pci.c:168)
librte_bus_pci.so.24!rte_pci_probe_one_driver(struct rte_pci_driver * dr,
struct rte_pci_device * dev) (\dpdk-23.11\drivers\bus\pci\pci_common.c:312)
librte_bus_pci.so.24!pci_probe_all_drivers(struct rte_pci_device * dev)
(\dpdk-23.11\drivers\bus\pci\pci_common.c:396)
librte_bus_pci.so.24!pci_probe()
(\dpdk-23.11\drivers\bus\pci\pci_common.c:423)
librte_eal.so.24!rte_bus_probe()
(\dpdk-23.11\lib\eal\common\eal_common_bus.c:78)
librte_eal.so.24!rte_eal_init(int argc, char ** argv)
(\dpdk-23.11\lib\eal\linux\eal.c:1287)

Best regards,
Julien Marcin

[-- Attachment #2: Type: text/html, Size: 6122 bytes --]

Re: [mlx5 driver] Usage of mlx5 with unpriviledged LXC container

[David Marchand]

On Mon, 25 Aug 2025 at 14:09, Julien <julien.marcin.tech@gmail.com> wrote:
>
> Hello,
> I have a question about using the mlx5 driver with LXC.
> I'm trying to use dpdk-testpmd in an LXC container whose root user isn't mapped to the host's root user.
> Note: The entire physical interface is given to the LXC container, not a virtual interface.
>
> The following error occured:
> mlx5_common: DevX create TIS failed errno=22 status=0 syndrome=0
> mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_2.
> mlx5_net: TIS allocation failure
> mlx5_net: probe of PCI device 0000:27:00.0 aborted after encountering an error: Cannot allocate memory
> mlx5_common: Failed to load driver mlx5_eth
> EAL: Requested device 0000:27:00.0 cannot be used
> EAL: Bus (pci) probe failed.
>
> The "transport_domain" is created, and the mlx5_devx_cmd_create_td() function runs normally.
> The call to mlx5dv_devx_obj_create() receives an errno of 22.
>
> I don't encounter any problems when the container's root user is mapped to the host's root user.
> Has anyone experienced this before?
> Is it possible to use the driver in an unprivileged LXC container?

There is probably something missing in terms of capabilities.
I don't know how LXC behaves in this regard.

I suggest you look at "5.5.1.5. Run as Non-Root" in
https://doc.dpdk.org/guides/platform/mlx5.html.


-- 
David Marchand

Re: [mlx5 driver] Usage of mlx5 with unpriviledged LXC container

[Julien]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

Thank you for the response.
Yes, these capabilities are necessary in order to run dpdk-testpmd inside a
container. However, they are not sufficient.
I tried again with a 6.8 kernel and the Nvidia tools/libraries suite:
"doca-host_3.1.0-091000-25.07-ubuntu2404_amd64.deb".
The log is more informative this time: "mlx5_common: DevX create TIS failed
errno=121 status=0x3 syndrome=0x6a6678".
Do you have any idea how to investigate this issue, especially regarding
"status" and "syndrome"?

On Mon, Aug 25, 2025 at 2:24 PM David Marchand <david.marchand@redhat.com>
wrote:

> On Mon, 25 Aug 2025 at 14:09, Julien <julien.marcin.tech@gmail.com> wrote:
> >
> > Hello,
> > I have a question about using the mlx5 driver with LXC.
> > I'm trying to use dpdk-testpmd in an LXC container whose root user isn't
> mapped to the host's root user.
> > Note: The entire physical interface is given to the LXC container, not a
> virtual interface.
> >
> > The following error occured:
> > mlx5_common: DevX create TIS failed errno=22 status=0 syndrome=0
> > mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_2.
> > mlx5_net: TIS allocation failure
> > mlx5_net: probe of PCI device 0000:27:00.0 aborted after encountering an
> error: Cannot allocate memory
> > mlx5_common: Failed to load driver mlx5_eth
> > EAL: Requested device 0000:27:00.0 cannot be used
> > EAL: Bus (pci) probe failed.
> >
> > The "transport_domain" is created, and the mlx5_devx_cmd_create_td()
> function runs normally.
> > The call to mlx5dv_devx_obj_create() receives an errno of 22.
> >
> > I don't encounter any problems when the container's root user is mapped
> to the host's root user.
> > Has anyone experienced this before?
> > Is it possible to use the driver in an unprivileged LXC container?
>
> There is probably something missing in terms of capabilities.
> I don't know how LXC behaves in this regard.
>
> I suggest you look at "5.5.1.5. Run as Non-Root" in
> https://doc.dpdk.org/guides/platform/mlx5.html.
>
>
> --
> David Marchand
>
>

[-- Attachment #2: Type: text/html, Size: 2835 bytes --]

Re: [mlx5 driver] Usage of mlx5 with unpriviledged LXC container

[David Marchand]

On Wed, 3 Sept 2025 at 16:54, Julien <julien.marcin.tech@gmail.com> wrote:
>
> Thank you for the response.
> Yes, these capabilities are necessary in order to run dpdk-testpmd inside a container. However, they are not sufficient.
> I tried again with a 6.8 kernel and the Nvidia tools/libraries suite: "doca-host_3.1.0-091000-25.07-ubuntu2404_amd64.deb".
> The log is more informative this time: "mlx5_common: DevX create TIS failed errno=121 status=0x3 syndrome=0x6a6678".
> Do you have any idea how to investigate this issue, especially regarding "status" and "syndrome"?

Sorry, I can't help about such low level details for the mlx5 driver.
I copied the DPDK mlx5 maintainers, maybe they can help.

However, I am a bit concerned as you are mixing in some DOCA
proprietary libraries.
If you need DOCA, you should probably contact the NVIDIA DOCA support team.


-- 
David Marchand

RE: [mlx5 driver] Usage of mlx5 with unpriviledged LXC container

[Asaf Penso]

+ @Maayan Kashani, please help here

Regards,
Asaf Penso

>-----Original Message-----
>From: David Marchand <david.marchand@redhat.com>
>Sent: Wednesday, 3 September 2025 18:07
>To: Julien <julien.marcin.tech@gmail.com>
>Cc: users@dpdk.org; Dariusz Sosnowski <dsosnowski@nvidia.com>; Slava
>Ovsiienko <viacheslavo@nvidia.com>; Bing Zhao <bingz@nvidia.com>; Ori
>Kam <orika@nvidia.com>; Suanming Mou <suanmingm@nvidia.com>; Matan
>Azrad <matan@nvidia.com>
>Subject: Re: [mlx5 driver] Usage of mlx5 with unpriviledged LXC container
>
>On Wed, 3 Sept 2025 at 16:54, Julien <julien.marcin.tech@gmail.com> wrote:
>>
>> Thank you for the response.
>> Yes, these capabilities are necessary in order to run dpdk-testpmd inside a
>container. However, they are not sufficient.
>> I tried again with a 6.8 kernel and the Nvidia tools/libraries suite: "doca-
>host_3.1.0-091000-25.07-ubuntu2404_amd64.deb".
>> The log is more informative this time: "mlx5_common: DevX create TIS failed
>errno=121 status=0x3 syndrome=0x6a6678".
>> Do you have any idea how to investigate this issue, especially regarding
>"status" and "syndrome"?
>
>Sorry, I can't help about such low level details for the mlx5 driver.
>I copied the DPDK mlx5 maintainers, maybe they can help.
>
>However, I am a bit concerned as you are mixing in some DOCA proprietary
>libraries.
>If you need DOCA, you should probably contact the NVIDIA DOCA support
>team.
>
>
>--
>David Marchand