mlx5_stats_get reports crash: invalid ibv_context

mlx5_stats_get reports crash: invalid ibv_context

[narsimharaj pentam]

[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]

Hi Team,


A crash is observed on  MLX5 device  while  fetching stats "mlx5_stats_get"
 , looks like *ibv_context*  is getting corrupted  ( ibctx=0x1ede838800001e)



We do access the stats from multiple threads  simultaneously , does this
create any issue ?



Platform details:



Platform      :     VM

Hypervisor    :     Linux KVM

Cloud         :     None

Application   :     64-bit

Datapath      :     DPDK

DPDK : dpdk-22.11.1

Library : libmlx5-rdmav34.so





Program terminated with signal 11, Segmentation fault.

#0  0x00007ffff0114c83 in mlx5_get_dv_ops (ibctx=0x1ede838800001e) at
../providers/mlx5/mlx5.c:2734

2734    ../providers/mlx5/mlx5.c: No such file or directory.

(gdb) bt

#0  0x00007ffff0114c83 in mlx5_get_dv_ops (ibctx=0x1ede838800001e) at
../providers/mlx5/mlx5.c:2734

#1  0x00007ffff012f378 in mlx5dv_devx_obj_query (obj=0x27cabd0
<SipMedia::SipMedia(SipMedia const&)+1120>, in=0x7fffa6badb20, inlen=32,
out=0x7fffa6badb40, outlen=1024)

    at ../providers/mlx5/verbs.c:5808

#2  0x0000000004fc0eeb in mlx5_devx_cmd_queue_counter_query
(dcs=0x31131be2b2c0, clear=clear@entry=0,
out_of_buffers=out_of_buffers@entry=0x7fffa6bae010)

    at ../usbc_gnu_src/drivers/common/mlx5/mlx5_devx_cmds.c:2845

#3  0x00000000052ec96d in mlx5_os_read_dev_stat
(priv=priv@entry=0x31131bf66400,
ctr_name=<optimized out>, ctr_name@entry=0x5b6fcc2 "out_of_buffer",
stat=stat@entry=0x7fffa6bae010)

    at ../usbc_gnu_src/drivers/net/mlx5/linux/mlx5_os.c:2684

#4  0x00000000053719e4 in mlx5_stats_get (dev=<optimized out>,
stats=0x7fffa6bae690) at ../usbc_gnu_src/drivers/net/mlx5/mlx5_stats.c:155

#5  0x0000000005076873 in rte_eth_stats_get () at
../usbc_gnu_src/lib/ethdev/rte_ethdev.c:2733



BR

Narsimha

[-- Attachment #2: Type: text/html, Size: 5189 bytes --]

RE: mlx5_stats_get reports crash: invalid ibv_context

[Maayan Kashani]

[-- Attachment #1: Type: text/plain, Size: 2189 bytes --]

Hi, Narsimha,
Sorry for the late response,
in mlx5 stats we do mention that
- For secondary process:
  - "out_of_buffer" statistics is not supported.

Do you have some testpmd reproduction?


Regards,
Maayan Kashani

From: narsimharaj pentam <pnarsimharaj@gmail.com>
Sent: Wednesday, 28 February 2024 6:32
To: users@dpdk.org
Subject: mlx5_stats_get reports crash: invalid ibv_context

External email: Use caution opening links or attachments

Hi Team,

A crash is observed on  MLX5 device  while  fetching stats "mlx5_stats_get"  , looks like ibv_context  is getting corrupted  ( ibctx=0x1ede838800001e)

We do access the stats from multiple threads  simultaneously , does this create any issue ?

Platform details:

Platform      :     VM
Hypervisor    :     Linux KVM
Cloud         :     None
Application   :     64-bit
Datapath      :     DPDK
DPDK : dpdk-22.11.1
Library : libmlx5-rdmav34.so


Program terminated with signal 11, Segmentation fault.
#0  0x00007ffff0114c83 in mlx5_get_dv_ops (ibctx=0x1ede838800001e) at ../providers/mlx5/mlx5.c:2734
2734    ../providers/mlx5/mlx5.c: No such file or directory.
(gdb) bt
#0  0x00007ffff0114c83 in mlx5_get_dv_ops (ibctx=0x1ede838800001e) at ../providers/mlx5/mlx5.c:2734
#1  0x00007ffff012f378 in mlx5dv_devx_obj_query (obj=0x27cabd0 <SipMedia::SipMedia(SipMedia const&)+1120>, in=0x7fffa6badb20, inlen=32, out=0x7fffa6badb40, outlen=1024)
    at ../providers/mlx5/verbs.c:5808
#2  0x0000000004fc0eeb in mlx5_devx_cmd_queue_counter_query (dcs=0x31131be2b2c0, clear=clear@entry=0, out_of_buffers=out_of_buffers@entry=0x7fffa6bae010)
    at ../usbc_gnu_src/drivers/common/mlx5/mlx5_devx_cmds.c:2845
#3  0x00000000052ec96d in mlx5_os_read_dev_stat (priv=priv@entry=0x31131bf66400, ctr_name=<optimized out>, ctr_name@entry=0x5b6fcc2 "out_of_buffer", stat=stat@entry=0x7fffa6bae010)
    at ../usbc_gnu_src/drivers/net/mlx5/linux/mlx5_os.c:2684
#4  0x00000000053719e4 in mlx5_stats_get (dev=<optimized out>, stats=0x7fffa6bae690) at ../usbc_gnu_src/drivers/net/mlx5/mlx5_stats.c:155
#5  0x0000000005076873 in rte_eth_stats_get () at ../usbc_gnu_src/lib/ethdev/rte_ethdev.c:2733

BR
Narsimha

[-- Attachment #2: Type: text/html, Size: 10493 bytes --]

RE: mlx5_stats_get reports crash: invalid ibv_context

[Bing Zhao]

[-- Attachment #1: Type: text/plain, Size: 2696 bytes --]

Hi @narsimharaj pentam<mailto:pnarsimharaj@gmail.com>,
Is it multiple threads or multiple processes as the secondary as Maayan mentioned? BTW, is it a bond configuration?

From: Maayan Kashani <mkashani@nvidia.com>
Sent: Wednesday, December 4, 2024 3:36 PM
To: narsimharaj pentam <pnarsimharaj@gmail.com>; users@dpdk.org
Subject: RE: mlx5_stats_get reports crash: invalid ibv_context

External email: Use caution opening links or attachments

Hi, Narsimha,
Sorry for the late response,
in mlx5 stats we do mention that
- For secondary process:
  - "out_of_buffer" statistics is not supported.

Do you have some testpmd reproduction?


Regards,
Maayan Kashani

From: narsimharaj pentam <pnarsimharaj@gmail.com<mailto:pnarsimharaj@gmail.com>>
Sent: Wednesday, 28 February 2024 6:32
To: users@dpdk.org<mailto:users@dpdk.org>
Subject: mlx5_stats_get reports crash: invalid ibv_context

External email: Use caution opening links or attachments

Hi Team,

A crash is observed on  MLX5 device  while  fetching stats "mlx5_stats_get"  , looks like ibv_context  is getting corrupted  ( ibctx=0x1ede838800001e)

We do access the stats from multiple threads  simultaneously , does this create any issue ?

Platform details:

Platform      :     VM
Hypervisor    :     Linux KVM
Cloud         :     None
Application   :     64-bit
Datapath      :     DPDK
DPDK : dpdk-22.11.1
Library : libmlx5-rdmav34.so


Program terminated with signal 11, Segmentation fault.
#0  0x00007ffff0114c83 in mlx5_get_dv_ops (ibctx=0x1ede838800001e) at ../providers/mlx5/mlx5.c:2734
2734    ../providers/mlx5/mlx5.c: No such file or directory.
(gdb) bt
#0  0x00007ffff0114c83 in mlx5_get_dv_ops (ibctx=0x1ede838800001e) at ../providers/mlx5/mlx5.c:2734
#1  0x00007ffff012f378 in mlx5dv_devx_obj_query (obj=0x27cabd0 <SipMedia::SipMedia(SipMedia const&)+1120>, in=0x7fffa6badb20, inlen=32, out=0x7fffa6badb40, outlen=1024)
    at ../providers/mlx5/verbs.c:5808
#2  0x0000000004fc0eeb in mlx5_devx_cmd_queue_counter_query (dcs=0x31131be2b2c0, clear=clear@entry=0, out_of_buffers=out_of_buffers@entry=0x7fffa6bae010)
    at ../usbc_gnu_src/drivers/common/mlx5/mlx5_devx_cmds.c:2845
#3  0x00000000052ec96d in mlx5_os_read_dev_stat (priv=priv@entry=0x31131bf66400, ctr_name=<optimized out>, ctr_name@entry=0x5b6fcc2 "out_of_buffer", stat=stat@entry=0x7fffa6bae010)
    at ../usbc_gnu_src/drivers/net/mlx5/linux/mlx5_os.c:2684
#4  0x00000000053719e4 in mlx5_stats_get (dev=<optimized out>, stats=0x7fffa6bae690) at ../usbc_gnu_src/drivers/net/mlx5/mlx5_stats.c:155
#5  0x0000000005076873 in rte_eth_stats_get () at ../usbc_gnu_src/lib/ethdev/rte_ethdev.c:2733

BR
Narsimha

[-- Attachment #2: Type: text/html, Size: 12505 bytes --]