From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 13C55456A5 for ; Wed, 24 Jul 2024 18:08:03 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B1C4E434F3; Wed, 24 Jul 2024 18:08:02 +0200 (CEST) Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by mails.dpdk.org (Postfix) with ESMTP id 45FCD40B97 for ; Wed, 24 Jul 2024 18:07:56 +0200 (CEST) Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-79f19f19059so397507885a.0 for ; Wed, 24 Jul 2024 09:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721837275; x=1722442075; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=XZh4yXsLDLz3f14x0XuqW4M/8qj8m5HtIS+urH6CFIA=; b=RI6C4HEOlnFUgCCdEDLtIUgdaNHUTnsTmErs4B/yipqyLGuEo/USkin00JDg7K4LkX FZKHGH8VXm38dcqrO9aEwg07pZ1LsfnHvcoq2QXS+Q/8vN1zHkbQBCMyxchNm10CEx6u q7bc0/Xv0AIFfBriqb6ivOQx55+VHam1p/DT4sSwKjRDj5FyVuLhIKKvJOlf+tUUbbvq wK3nkC2zxuHtqRF1z4fskF+l3FprJNP4jCYHUnPVh7IUfP5ULADBp58HtixkLGCbfiI1 Mnayqpge3LRkrBFkQaUMDwJZTQi9ICNuTuVTcRsxzyFtwpiekHzo3sSdlCjEym8FthLr Cf7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721837275; x=1722442075; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XZh4yXsLDLz3f14x0XuqW4M/8qj8m5HtIS+urH6CFIA=; b=Dzql8qek+SBpphZtkki7m4M1ltLWbUV+NHEOE5OlnTSCdD7ubeZIw2lCdxJiFaUspe sH9Ah1keSQmNFzTwDMwyE2l65IdIvHSbFeEniWj3QEs27ROHnpkbHb64bA5Zr5DB/3cY 25YB+nFya7+OTqL8TgAgmJHCdaCWiikLuiqmZELoz47JZ4NAymkpB431lJQB2ZlA4Lxr fHlsdcACC/biRMfFlHGAKM3FjZX3ASjKGgUMHGnGtBZpZQRR88WBTnX6mmxrxZeNrIBC ABXu7gf+OmR/qrjKcg/HKpbnWG55jVow8AA2kphYvfX39yWqv7CDvibAQWn09FLt6bxI LyPg== X-Forwarded-Encrypted: i=1; AJvYcCVNQcJ+Q7M3Yh/2vuX4bwiueEdgNUTRoUJnTUtgb/G2CEWJ6tCWhkOb361FPMhWxroYmQo5SNvgjcV0bGU4cQ== X-Gm-Message-State: AOJu0Yx7XGYJeT0qSzj2D3gHkPt51Nj/6DAZlj7F/NR3zmMZ1NY86z2k x5nUt99hngkcqSSSKMO3WbEQpAT/9ofSySd7SajWVj8bvmqUsWbgLLK/WHw6EtzFvlz4z8zXlqx ARtRkHWlFEbNjb5gKjykHjJv1EAQ= X-Google-Smtp-Source: AGHT+IEk13YKdBjCBF4NQsydqeGNdZWOdiaj1jzu2PfKaM+VlyY0efvgl1ML5Q/8hPUj8IlOKbbJVqqk/JFZuez7yhs= X-Received: by 2002:a05:620a:4721:b0:79f:6b2:dd0 with SMTP id af79cd13be357-7a1d4511367mr16321485a.27.1721837275316; Wed, 24 Jul 2024 09:07:55 -0700 (PDT) MIME-Version: 1.0 References: <5485686.Sb9uPGUboI@thomas> In-Reply-To: From: Yasuhiro Ohara Date: Thu, 25 Jul 2024 01:07:44 +0900 Message-ID: Subject: Re: ConnectX-7 400GbE wire-rate? To: Cliff Burdick Cc: Thomas Monjalon , users@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Cliff. I saw it, and thought at that time that 200GbE x 2 might be different from 400GbE x 1. But good to know that you hit 400Gbps multiple times. Thank you. regards, Yasu 2024=E5=B9=B47=E6=9C=8824=E6=97=A5(=E6=B0=B4) 23:50 Cliff Burdick : > > To answer your original question, yes, I've hit 400Gbps many times provid= ed there were enough queues/cores. You can also see test 12.2 here achieved= line rate: > > https://fast.dpdk.org/doc/perf/DPDK_23_11_NVIDIA_NIC_performance_report.p= df > > On Wed, Jul 24, 2024 at 4:58=E2=80=AFAM Yasuhiro Ohara wrote: >> >> Hi Thomas, >> Thank you for getting back to this. >> >> From what we have seen, it seemed to be rather a Mellanox firmware issue >> than the pktgen or DPDK issue. >> When we were using ConnectX-7 with fw ver 28.41.1000 >> on a Core i9 machine, it limited its tx-bandwidth to 100Gbps, >> though the physical max should be 400Gbps. >> If we lower the fw version to 28.39.2048 or 28.39.3004, >> it could send 256Gbps, on the same core i9 machine. >> >> Our bandwidth demonstration event completed successfully, >> so we are fine with the small knowledge. >> I think my colleague can help if someone wants to debug >> further about this issue. >> >> Thank you anyway. >> >> Regards, >> Yasu >> >> 2024=E5=B9=B47=E6=9C=8823=E6=97=A5(=E7=81=AB) 2:37 Thomas Monjalon : >> > >> > Hello, >> > >> > I see there is no answer. >> > >> > Did you try with testpmd? >> > I don't know whether it can be a limitation of dpdk-pktgen? >> > >> > >> > >> > 29/05/2024 03:32, Yasuhiro Ohara: >> > > Hi. My colleague is trying to generate 400Gbps traffic using Connect= X-7, >> > > but with no luck. >> > > >> > > Has anyone succeeded in generating 400Gbps (by larger than 1500B is = ok), >> > > or are there any known issues? >> > > >> > > Using dpdk-pktgen, the link is successfully recognized as 400GbE >> > > but when we start the traffic, it seems to be capped at 100Gbps. >> > > >> > > Some info follows. >> > > >> > > [ 1.490256] mlx5_core 0000:01:00.0: firmware version: 28.41.1000 >> > > [ 1.492853] mlx5_core 0000:01:00.0: 504.112 Gb/s available PCIe >> > > bandwidth (32.0 GT/s PCIe x16 link) >> > > [ 1.805419] mlx5_core 0000:01:00.0: Rate limit: 127 rates are >> > > supported, range: 0Mbps to 195312Mbps >> > > [ 1.808477] mlx5_core 0000:01:00.0: E-Switch: Total vports 18, pe= r >> > > vport: max uc(128) max mc(2048) >> > > [ 1.827270] mlx5_core 0000:01:00.0: Port module event: module 0, >> > > Cable unplugged >> > > [ 1.830317] mlx5_core 0000:01:00.0: mlx5_pcie_event:298:(pid 9): >> > > PCIe slot advertised sufficient power (75W). >> > > [ 1.830610] mlx5_core 0000:01:00.0: MLX5E: StrdRq(1) RqSz(8) >> > > StrdSz(2048) RxCqeCmprss(0) >> > > [ 1.929813] mlx5_core 0000:01:00.0: Supported tc offload range - >> > > chains: 4294967294, prios: 4294967295 >> > > >> > > Device type: ConnectX7 >> > > Name: MCX75310AAS-NEA_Ax >> > > Description: NVIDIA ConnectX-7 HHHL Adapter card; 400GbE / NDR IB >> > > (default mode); Single-port OSFP; PCIe 5.0 x16; Crypto Disabled; >> > > Secure Boot Enabled; >> > > Device: /dev/mst/mt4129_pciconf0 >> > > >> > > Enabled Link Speed (Ext.) : 0x00010000 (400G_4X) >> > > Supported Cable Speed (Ext.) : 0x00010800 (400G_4X,100G_1X) >> > > >> > > pktgen result: >> > > >> > > Pkts/s Rx : 0 >> > > TX : 8059264 >> > > >> > > MBits/s Rx/Tx : 0/97742 >> > > >> > > >> > > Any info is appreciated. Thanks. >> > > >> > > regards, >> > > Yasu >> > > >> > >> > >> > >> > >> >