From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3F21243A46; Wed, 7 Feb 2024 13:40:23 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0EDE1402EA; Wed, 7 Feb 2024 13:40:23 +0100 (CET) Received: from mail-ed1-f43.google.com (mail-ed1-f43.google.com [209.85.208.43]) by mails.dpdk.org (Postfix) with ESMTP id 7D41040279 for ; Wed, 7 Feb 2024 13:40:21 +0100 (CET) Received: by mail-ed1-f43.google.com with SMTP id 4fb4d7f45d1cf-560a9738081so627155a12.1 for ; Wed, 07 Feb 2024 04:40:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netgate.com; s=google; t=1707309621; x=1707914421; darn=dpdk.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=A7Xw4zpN9yUTG+TDWrh16ByB2EqjSsvAsK4QUWcjjfw=; b=dtG/ZhOQuxgmAuFT9+kQago6d6TV9DArDUqqOSMEdujhQtJxm2g5AZLosU5QOInW7d T2sZP/Ju2IjpIYolcbYV4/lsybHcIIXbKX1kRL7YNQB9YXDHArWAeMkRYIoshn2Tj7Oa WnC8w5r1kgYDB5kGC69SX3YzQLGTFYy9NFvxM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707309621; x=1707914421; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=A7Xw4zpN9yUTG+TDWrh16ByB2EqjSsvAsK4QUWcjjfw=; b=qkAAvjiVYB/sfxPlmV19rgcbc3bXRbvSQSwf4/RuebJNp0xWpVKNox7s5wnQWX4aU7 uK9kSXwp/dKKKhr2eurDXWC4j5hVjFb2qWaiZzXaJdw98PaDz7btW6hb71G/1V8BuFu4 R8J4ScREusFv2OK3sFmhFlgTyiujDES11H+o0jES8S3wfjMjfnD6CLRBdqZACwFB1//r 4bKjx0uNw9g0tCU+ghSw6gw1cp8o7e9pQ0cE3OJ1hL5DQZRah3lDbkV9rh4QiVKNOlQd Pac6foCQekDWfv5rQD4MMz/PZAY/rKQ/6q+SfRplCuKT5SkEjUVxacdQpSmCSFc9zMJN 3tqQ== X-Gm-Message-State: AOJu0Yy+IkW77nPISIxdUrgukT6fzpj3NPkCUaxbbliOCO/UubYmDKfC 0K2PWMx9nfhoKTjtGinahWfLDNDLSrjtHnCaPncW+z+xjXa5CDecNQOALxJQSZnYabeU8nmn9pv wO+dtBIy4uBNq9aVA71Bkii9U6erAIaJgHj8vbVuvm7e1158XGHRQ X-Google-Smtp-Source: AGHT+IGrFW9ZXC1qCbQ3EVeODUw7k1AhSl0ElmpC+rorOJLm2jVFLC1VBL7pFMYPzPkSMW/8dB7I+vwFt3uet/P8Sa8= X-Received: by 2002:aa7:c0d5:0:b0:560:ca08:59d8 with SMTP id j21-20020aa7c0d5000000b00560ca0859d8mr1441252edp.36.1707309620697; Wed, 07 Feb 2024 04:40:20 -0800 (PST) MIME-Version: 1.0 From: Vladimir Ratnikov Date: Wed, 7 Feb 2024 13:40:09 +0100 Message-ID: Subject: Azure(hyperv) hugepages issues with Mellanox NICs(mlx5) To: dev@dpdk.org Content-Type: multipart/alternative; boundary="00000000000076d0bb0610c9ff1e" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --00000000000076d0bb0610c9ff1e Content-Type: text/plain; charset="UTF-8" Hello! We observe the problem with hugepages on Azure environment with mellanox devices there(Currently MT27710 Connect-X 4) which use mlx5 PMD. We use a DPDK based application, but when using testpmd for debug purposes we observe exact the same issue. After each restart of the process, 2 hugepages(2 Mellanox NICs) disappear from the pool until they're totally exhausted. Checking using /proc/meminfo. That's very weird behavior. If user space application exits, hugepages will be freed in any case(except in situations where files are being kept on hugetlb FS). So it probably tells that maybe some kernel process holds those pages(tried to rmmod/modprobe all related modules and it didn't help. On hugetlb FS there's nothing at this point) Found that one of the places where the issue appears to happen is mlx5_malloc function when ~2MB of memory is being allocated while creating the device. Stack trace is: > #0 mlx5_malloc (flags=4, size=2097168, align=64, socket=-1) at > ../src-dpdk/drivers/common/mlx5/mlx5_malloc.c:174 > #1 0x00007fffae258cad in _mlx5_ipool_malloc_cache (pool=0xac036ac40, > cidx=0, idx=0x7fffa8759e90) at ../src-dpdk/drivers/net/mlx5/mlx5_utils.c:410 > #2 0x00007fffae258e42 in mlx5_ipool_malloc_cache (pool=0xac036ac40, > idx=0x7fffa8759e90) at ../src-dpdk/drivers/net/mlx5/mlx5_utils.c:441 > #3 0x00007fffae259208 in mlx5_ipool_malloc (pool=0xac036ac40, > idx=0x7fffa8759e90) at ../src-dpdk/drivers/net/mlx5/mlx5_utils.c:521 > #4 0x00007fffae2593d0 in mlx5_ipool_zmalloc (pool=0xac036ac40, > idx=0x7fffa8759e90) at ../src-dpdk/drivers/net/mlx5/mlx5_utils.c:575 > #5 0x00007fffabbb1b73 in flow_dv_discover_priorities (dev=0x7fffaf97ae80 > , vprio=0x7fffaf1e492e , vprio_n=2) at > ../src-dpdk/drivers/net/mlx5/mlx5_flow_dv.c:19706 > #6 0x00007fffabb4d126 in mlx5_flow_discover_priorities > (dev=0x7fffaf97ae80 ) at > ../src-dpdk/drivers/net/mlx5/mlx5_flow.c:11963 > #7 0x00007fffae52cf48 in mlx5_dev_spawn (dpdk_dev=0x55555575bca0, > spawn=0xac03d7340, eth_da=0x7fffa875a1d0, mkvlist=0x0) at > ../src-dpdk/drivers/net/mlx5/linux/mlx5_os.c:1649 > #8 0x00007fffae52f1a3 in mlx5_os_pci_probe_pf (cdev=0xac03d9d80, > req_eth_da=0x7fffa875a300, owner_id=0, mkvlist=0x0) at > ../src-dpdk/drivers/net/mlx5/linux/mlx5_os.c:2348 > #9 0x00007fffae52f91f in mlx5_os_pci_probe (cdev=0xac03d9d80, > mkvlist=0x0) at ../src-dpdk/drivers/net/mlx5/linux/mlx5_os.c:2497 > #10 0x00007fffae52fd09 in mlx5_os_net_probe (cdev=0xac03d9d80, > mkvlist=0x0) at ../src-dpdk/drivers/net/mlx5/linux/mlx5_os.c:2578 > #11 0x00007fffa93f297b in drivers_probe (cdev=0xac03d9d80, user_classes=1, > mkvlist=0x0) at ../src-dpdk/drivers/common/mlx5/mlx5_common.c:937 > #12 0x00007fffa93f2c95 in mlx5_common_dev_probe (eal_dev=0x55555575bca0) > at ../src-dpdk/drivers/common/mlx5/mlx5_common.c:1027 > #13 0x00007fffa94105b3 in mlx5_common_pci_probe (pci_drv=0x7fffaf492680 > , pci_dev=0x55555575bc90) at > ../src-dpdk/drivers/common/mlx5/mlx5_common_pci.c:168 > #14 0x00007fffa9297950 in rte_pci_probe_one_driver (dr=0x7fffaf492680 > , dev=0x55555575bc90) at > ../src-dpdk/drivers/bus/pci/pci_common.c:312 > #15 0x00007fffa9297be4 in pci_probe_all_drivers (dev=0x55555575bc90) at > ../src-dpdk/drivers/bus/pci/pci_common.c:396 > #16 0x00007fffa9297c6d in pci_probe () at > ../src-dpdk/drivers/bus/pci/pci_common.c:423 > #17 0x00007fffa9c8f551 in rte_bus_probe () at > ../src-dpdk/lib/eal/common/eal_common_bus.c:78 > #18 0x00007fffa9cd80c2 in rte_eal_init (argc=7, argv=0x7fffbba74ff8) at > ../src-dpdk/lib/eal/linux/eal.c:1300 We found that there's a devarg for mlx5 PMD `sys_mem_en=1` which allows using system memory instead of RTE memory. It helped a bit. Now just one hugepage is missing after each restart. Also tried to reproduce the same on hardware and it's fine there(a bit different NICs MT27800, but using the same PMD mlx5). So our thoughts that something could be related somehow with Azure(hyperv?) environment. So wondering if someone observed the same issue? Many thanks! --00000000000076d0bb0610c9ff1e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello!

We observe the problem with hugepages on Azu= re environment with mellanox devices there(Currently MT27710 Connect-X 4) w= hich use mlx5 PMD.
We use a DPDK based application, but when using testp= md for debug purposes we observe exact the same issue. After each restart o= f the process, 2 hugepages(2 Mellanox NICs) disappear from the pool until t= hey're totally exhausted.
Checking using /proc/meminfo.

That&= #39;s very weird behavior. If user space application exits, hugepages will = be freed in any case(except in situations where files are being kept on hug= etlb FS).
So it probably tells that maybe some kernel process holds thos= e pages(tried to rmmod/modprobe all related modules and it didn't help.= On hugetlb FS there's nothing at this point)

Found that one of = the places where the issue appears to happen is mlx5_malloc function when ~= 2MB of memory is being allocated while creating the device.

Stack tr= ace is:
#0 =C2=A0mlx5_= malloc (flags=3D4, size=3D2097168, align=3D64, socket=3D-1) at ../src-dpdk/= drivers/common/mlx5/mlx5_malloc.c:174
#1 =C2=A00x00007fffae258cad in _ml= x5_ipool_malloc_cache (pool=3D0xac036ac40, cidx=3D0, idx=3D0x7fffa8759e90) = at ../src-dpdk/drivers/net/mlx5/mlx5_utils.c:410
#2 =C2=A00x00007fffae25= 8e42 in mlx5_ipool_malloc_cache (pool=3D0xac036ac40, idx=3D0x7fffa8759e90) = at ../src-dpdk/drivers/net/mlx5/mlx5_utils.c:441
#3 =C2=A00x00007fffae25= 9208 in mlx5_ipool_malloc (pool=3D0xac036ac40, idx=3D0x7fffa8759e90) at ../= src-dpdk/drivers/net/mlx5/mlx5_utils.c:521
#4 =C2=A00x00007fffae2593d0 i= n mlx5_ipool_zmalloc (pool=3D0xac036ac40, idx=3D0x7fffa8759e90) at ../src-d= pdk/drivers/net/mlx5/mlx5_utils.c:575
#5 =C2=A00x00007fffabbb1b73 in flo= w_dv_discover_priorities (dev=3D0x7fffaf97ae80 <rte_eth_devices>, vpr= io=3D0x7fffaf1e492e <vprio>, vprio_n=3D2) at ../src-dpdk/drivers/net/= mlx5/mlx5_flow_dv.c:19706
#6 =C2=A00x00007fffabb4d126 in mlx5_flow_disco= ver_priorities (dev=3D0x7fffaf97ae80 <rte_eth_devices>) at ../src-dpd= k/drivers/net/mlx5/mlx5_flow.c:11963
#7 =C2=A00x00007fffae52cf48 in mlx5= _dev_spawn (dpdk_dev=3D0x55555575bca0, spawn=3D0xac03d7340, eth_da=3D0x7fff= a875a1d0, mkvlist=3D0x0) at ../src-dpdk/drivers/net/mlx5/linux/mlx5_os.c:16= 49
#8 =C2=A00x00007fffae52f1a3 in mlx5_os_pci_probe_pf (cdev=3D0xac03d9d= 80, req_eth_da=3D0x7fffa875a300, owner_id=3D0, mkvlist=3D0x0) at ../src-dpd= k/drivers/net/mlx5/linux/mlx5_os.c:2348
#9 =C2=A00x00007fffae52f91f in m= lx5_os_pci_probe (cdev=3D0xac03d9d80, mkvlist=3D0x0) at ../src-dpdk/drivers= /net/mlx5/linux/mlx5_os.c:2497
#10 0x00007fffae52fd09 in mlx5_os_net_pro= be (cdev=3D0xac03d9d80, mkvlist=3D0x0) at ../src-dpdk/drivers/net/mlx5/linu= x/mlx5_os.c:2578
#11 0x00007fffa93f297b in drivers_probe (cdev=3D0xac03d= 9d80, user_classes=3D1, mkvlist=3D0x0) at ../src-dpdk/drivers/common/mlx5/m= lx5_common.c:937
#12 0x00007fffa93f2c95 in mlx5_common_dev_probe (eal_de= v=3D0x55555575bca0) at ../src-dpdk/drivers/common/mlx5/mlx5_common.c:1027#13 0x00007fffa94105b3 in mlx5_common_pci_probe (pci_drv=3D0x7fffaf492680= <mlx5_common_pci_driver>, pci_dev=3D0x55555575bc90) at ../src-dpdk/d= rivers/common/mlx5/mlx5_common_pci.c:168
#14 0x00007fffa9297950 in rte_p= ci_probe_one_driver (dr=3D0x7fffaf492680 <mlx5_common_pci_driver>, de= v=3D0x55555575bc90) at ../src-dpdk/drivers/bus/pci/pci_common.c:312
#15 = 0x00007fffa9297be4 in pci_probe_all_drivers (dev=3D0x55555575bc90) at ../sr= c-dpdk/drivers/bus/pci/pci_common.c:396
#16 0x00007fffa9297c6d in pci_pr= obe () at ../src-dpdk/drivers/bus/pci/pci_common.c:423
#17 0x00007fffa9c= 8f551 in rte_bus_probe () at ../src-dpdk/lib/eal/common/eal_common_bus.c:78=
#18 0x00007fffa9cd80c2 in rte_eal_init (argc=3D7, argv=3D0x7fffbba74ff8= ) at ../src-dpdk/lib/eal/linux/eal.c:1300

We found that the= re's a devarg for mlx5 PMD `sys_mem_en=3D1` which allows using system m= emory instead of RTE memory. It helped a bit. Now just one hugepage is miss= ing after each restart.
Also tried to reproduce the same on hardware and= it's fine there(a bit different NICs MT27800, but using the same PMD m= lx5).

So our thoughts that something could be related somehow with A= zure(hyperv?) environment. So wondering if someone observed the same issue?=

Many thanks!
--00000000000076d0bb0610c9ff1e--