From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 46746A2EEB for ; Mon, 7 Oct 2019 23:11:00 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6A7E11C11C; Mon, 7 Oct 2019 23:10:59 +0200 (CEST) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60061.outbound.protection.outlook.com [40.107.6.61]) by dpdk.org (Postfix) with ESMTP id 480781C202 for ; Fri, 4 Oct 2019 20:28:12 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZjnueJn6rgbGki1LO4TqJv4jjMcCdWslA1yRsfxxcd0VnoQtRQplI8FsK5dU5VUL2HArr+ZuoriALtpti+O32RIC4O+bQN2X4ka93OlY49J84WoiWJuL1MF5sj8QDrpesSO70Ss/W7CYzSA3K51GwgmSpw6Bq6JpTjt3BvVvV3oto8JBbmkYbMqviKX2pKE4b5dI3bGB+JeWCg5x/6C2KqPTydvfG2CL0800xaW33aCJGl5vPclkc+fdYeWGmNGlq/AL0P3N5ifJ45zPajEKrec7fMme93CskeQifNG5W1QgJePzjHvD+yHaYcb9Nd7glfoE0GJRVgA4rtOPLJ7OvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rQwxGEH6eq7IVcVSDY1LxzWBGGTs8b4R4enADF5jXJk=; b=WqSM39R8Hj81afBfrzYWi7d8es4gglCUqUgrzGnXDc6RIBUTnD6tNkD35S+byXV7pS7n2rwwz2B6CGE+QWO6Qcpj8k31JpeZK3KSnIcwsPT/UoZN3ErfxJngStB/bWvGCpMiLoPvw6Bkg3ENVR5ojf1KjS9yWQ5MmEip/yaJQmbcNcXYoPJXsQthVtfnG1M9LVH/nTLxIrKQys7pbg9Ylt6r9aWGQNpiGZ4BhBDIOwpuKKzPXbhvomxxOVcVCOi/G8C7ZtC93YTcbW7UHlrOFkFR/Mzge53SJ6MZKjCDGE4L/VGY+dRXxwbnVIxAoNLWd06YAFYkNGe3yMrMOVF3Ag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rQwxGEH6eq7IVcVSDY1LxzWBGGTs8b4R4enADF5jXJk=; b=lhxMD9TShAEczAUUJS2rrBgrhzqehUDESVwdFABNeagOM1GmGaSYcNvUKhnBPsrDQToPnyZSxptNOE/+zl829Rv9W7UMTUyDL+5nRepflhdDLEKz+lMSW9lDUemb5nhJJ2SivGAm+fFbXKC/2AN1z2dz5pgYWuhTP+CRL4W2pEc= Received: from AM6PR05MB5910.eurprd05.prod.outlook.com (20.179.3.81) by AM6PR05MB4869.eurprd05.prod.outlook.com (20.177.34.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2305.20; Fri, 4 Oct 2019 18:28:10 +0000 Received: from AM6PR05MB5910.eurprd05.prod.outlook.com ([fe80::5959:5341:23af:b0c7]) by AM6PR05MB5910.eurprd05.prod.outlook.com ([fe80::5959:5341:23af:b0c7%3]) with mapi id 15.20.2305.023; Fri, 4 Oct 2019 18:28:10 +0000 From: Kiran Vedere To: Jim Vaigl , Asaf Penso , 'Stephen Hemminger' CC: "users@dpdk.org" , Erez Ferber , Olga Shern , Danny Vogel Thread-Topic: [dpdk-users] DPDK on Mellanox BlueField Ref Platform Thread-Index: AdVy9ZHo1HqoOeJURCCyk0eROU1EBQABnMsAAAPtiAAAU1unIAASMIgQAYqaHLAABBg+EA== Date: Fri, 4 Oct 2019 18:28:09 +0000 Message-ID: References: <004101d572f5$92d479d0$b87d6d70$@com> <20190924101802.273c25d9@hermes.lan> <000001d5730b$bc69ea30$353dbe90$@com> <005001d57ada$1cc52900$564f7b00$@com> In-Reply-To: <005001d57ada$1cc52900$564f7b00$@com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=kiranv@mellanox.com; x-originating-ip: [71.184.199.187] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 055948a2-d84a-43dd-1047-08d748f89bed x-ms-office365-filtering-ht: Tenant x-ms-traffictypediagnostic: AM6PR05MB4869:|AM6PR05MB4869:|AM6PR05MB4869: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 018093A9B5 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(396003)(136003)(366004)(376002)(39860400002)(346002)(51914003)(129404003)(199004)(189003)(13464003)(6436002)(6116002)(71190400001)(7736002)(86362001)(478600001)(3846002)(305945005)(14454004)(71200400001)(9686003)(2906002)(55016002)(4326008)(6246003)(74316002)(186003)(26005)(229853002)(316002)(53546011)(6506007)(66066001)(25786009)(33656002)(110136005)(8676002)(5660300002)(54906003)(66946007)(81166006)(11346002)(107886003)(81156014)(446003)(99286004)(486006)(14444005)(256004)(76176011)(102836004)(7696005)(76116006)(66556008)(64756008)(66476007)(66446008)(8936002)(52536014)(476003); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR05MB4869; H:AM6PR05MB5910.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 58W4Y+m/W0itK2lMSTBR9oVwfPgjMqQhHNROsB2XldZq0gDrD42oIUAq64z2K7KALcOztGj3NJ226L/vDotRYP0r0nQHf5CxtpPB/snnAjarJ9KnwNC2IJ/bE7yyCBO4to1+l7PHM194D8U88NewK0FOpUqcrABWBNSMGTWmxL5BJ7uDYWwsmwpuQsq6dm8HOTjVNBVbFdzN6XG0Tq/0Tj86ziHDZkgyDmYRgvXkKqHgXSCEBfY8c8c6X+taUIvcWv3PUXL9v66O3xW7HCnGjF6TftpT10VYil1YqmuYO0oiZ/t1fuF6jF/t8rPO4yJPO4xUCvA54p13pPGiIKhVgkyWTbJeYTpaZYhnzyI19csNuZnvpFVdYsRRVn/EYoNUgyb/SwXw6Ah9K02aYxyF171LmUvUKjAk3UT+zZnBo2c= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 055948a2-d84a-43dd-1047-08d748f89bed X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Oct 2019 18:28:09.9489 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: DS1PXDuikDjh+OP7jVIGhcAdwVzzghtPYvNhjAn0mJknRTzQNFezvRqw4PPb/SBfbtlJ+AWTaJtaXsT1cesHuQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB4869 X-Mailman-Approved-At: Mon, 07 Oct 2019 23:10:58 +0200 Subject: Re: [dpdk-users] DPDK on Mellanox BlueField Ref Platform X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Jim, I tried your test with 9000 Byte MTU Size. On BlueField Reference Platform = I set the MTU of the interface to 9000 and on TRex I am sending 8096 size b= yte packets. I am able to loop back packets fine w/o any issues. Below is t= he command line I use for testpmd ./testpmd --log-level=3D"mlx5,8" -l 3,4,5,6,7,8,9,10,11,12,13,14,15 -n 4 -w= 17:00.0 --socket-mem=3D2048 -- --socket-num=3D0 --burst=3D64 --txd=3D2048 = --rxd=3D2048 --mbcache=3D512 --rxq=3D12 --txq=3D12 --nb-cores=3D12 -i -a --= forward-mode=3Dmac --max-pkt-len=3D9000 --mbuf-size=3D16384 Two things to consider: The max Rx packet len is used by the PMD during it= s Rx Queue initialization. By default this is set to 1518 Bytes for testpmd= /l3fwd. For jumbo frames you need to pass --max-pkt-len=3D9000 (for testpmd= ) or --enable-jumbo --max-pkt-len=3D9000 (for l3fwd). Are you passing these= values to l3fwd/testpmd when you run your test? Also since the mbuf_size i= s 2048 by default, you need to increase the mbuf_size to > Jumbo frame size= unless you enable scatter in the PMD. For testpmd you can increase the mbu= f size by using --mbuf-size parameter. For l3fwd I don't think there is a c= ommand line option to increase mbuf size in runtime. So you might need to r= ecompile the l3fwd code to increase mbuf size. Are you doing this? Hope this helps. Regards, Kiran -----Original Message----- From: Jim Vaigl =20 Sent: Friday, October 4, 2019 1:35 PM To: Asaf Penso ; 'Stephen Hemminger' Cc: users@dpdk.org; Kiran Vedere ; Erez Ferber ; Olga Shern ; Danny Vogel Subject: RE: [dpdk-users] DPDK on Mellanox BlueField Ref Platform A final update on this issue. Kiran Vedere went above and beyond the call = of duty: he completely reproduced my hardware setup, showed that it worked = using trex to generate similar traffic to mine, and then provided me with a= bundled-up .bfb of his CentOS (with updated kernel) and OFED install to tr= y so that there would be no configuration stuff for me to mess up. Using this, I saw exactly the same crashes I had seen in my setup. After some thought, I realized the only meaningful difference was that my t= raffic generator and IP configuration relied on an MTU size of 9000. Once I set the MTU size down to 1500, the crashes stopped. So, the answer is clearly that I'm just not setting up for the larger MTU s= ize. I need to start to understand how to get DPDK to manage that, but the= crashing is at least understood now, and I have a way forward. Thanks very much to Kiran. Regards, --Jim -----Original Message----- From: Jim Vaigl [mailto:jimv@rockbridgesoftware.com] Sent: Thursday, September 26, 2019 3:47 PM To: 'Asaf Penso'; 'Stephen Hemminger' Cc: 'users@dpdk.org'; 'Kiran Vedere'; 'Erez Ferber'; 'Olga Shern' Subject: RE: [dpdk-users] DPDK on Mellanox BlueField Ref Platform > From: Asaf Penso [mailto:asafp@mellanox.com] > Sent: Thursday, September 26, 2019 7:00 AM > To: Jim Vaigl; 'Stephen Hemminger' > Cc: users@dpdk.org; Kiran Vedere; Erez Ferber; Olga Shern > Subject: RE: [dpdk-users] DPDK on Mellanox BlueField Ref Platform > > Hello Jim, > > Thanks for your mail. > In order for us to have a better resolution please send a mail to our support team > - support@mellanox.com > Please provide as much info about the setup, configuration etc as you can= . > > In parallel, I added Erez Ferber here to assist. > > Regards, > Asaf Penso Thanks for the kind offer, Asaf. I'll take this debug effort off-line with= you and Erez and post back to the list here later with any resolution so e= veryone can see the result. By the way, the prior suggestion of using v. 25 of rdma-core didn't pan out= : the current build script just makes a local build in a subdirectory off = the source tree and there's no obvious way to integrate it with the MLNX_OF= ED environment and the dpdk install. After resolving package dependencies = to get rdma-core to build from the GitHub repo, I realized the instructions= say this: --- Building This project uses a cmake based build system. Quick start: $ bash build.sh build/bin will contain the sample programs and build/lib will contain the shared libraries. The build is configured to run all the programs 'in-place' and cannot be installed. NOTE: It is not currently easy to run from the build directory, the plugins only load from the system path. --- --Jim >> -----Original Message----- >> From: users On Behalf Of Jim Vaigl >> Sent: Tuesday, September 24, 2019 10:11 PM >> To: 'Stephen Hemminger' >> Cc: users@dpdk.org >> Subject: Re: [dpdk-users] DPDK on Mellanox BlueField Ref Platform >>=20 >> On Tue, 24 Sep 2019 12:31:51 -0400 >> "Jim Vaigl" wrote: >>=20 >>>> Since no one has chimed in with any build/install/configure=20 >>>> suggestion >> for >> >> the >> >> BlueField, I've spent some time debugging and thought I'd share=20 >> >> the >> results. >> >> Building the l3fwd example application and running it as the docs >> suggest, >> >> when >> >> I try to send it UDP packets from another machine, it dumps core. >> >> >> >> Debugging a bit with gdb and printf, I can see that from inside >> >> process_packet() >> >> and processx4_step1() the calls to rte_pktmbuf_mtod() return Nil=20 >> >> or suspicious pointer values (i.e. 0x80). The sample apps don't=20 >> >> guard against NULL pointers being returned from this rte call, so=20 >> >> that's why it's dumping core. >> >> >> >> I still think the problem is related to the driver config, but=20 >> >> thought >> this >> >> might ring a bell for anyone who's had problems like this. >> >> >> >> The thing that still bothers me is that rather than seeing what I=20 >> >> was expecting at init based on what the documentation shows: >> >> [...] >> >> EAL: probe driver: 15b3:1013 librte_pmd_mlx5 >> >> >> >> ... when rte_eal_init() runs, I'm seeing: >> >> [...] >> >> EAL: Selected IOVA mode 'PA' >> >> EAL: Probing VFIO support... >> >> >> >> This still seems wrong, and I've verified that specifying the BlueField >> >> target ID >> >> string in the make is causing "CONFIG_RTE_LIBRTE_MLX5_PMD=3Dy" to >> appear in >> >> the .config. >> >> >> >> Regards, >> >> --Jim Vaigl >> >> 614 886 5999 >> >> >> >> >> > >> >From: Stephen Hemminger [mailto:stephen@networkplumber.org] >> >Sent: Tuesday, September 24, 2019 1:18 PM >> >To: Jim Vaigl >> >Cc: users@dpdk.org >> > >> >Subject: Re: [dpdk-users] DPDK on Mellanox BlueField Ref Platform=20 >> >make sure you have latest version of rdma-core installed (v25). >> >The right version is not in most distros >>=20 >> Great suggestion. I'm using the rdma-core from the MLNX_OFED=20 >> 4.6-3.5.8.0 install. I can't figure out how to tell what version=20 >> that thing includes, >> even looking at the source, since there's no version information in=20 >> the source files, BUT I went to github and downloaded rdma-core v24=20 >> and v25 and neither diff cleanly with the source RPM that comes in=20 >> the OFED install. I don't know yet if it's because this is some=20 >> different version or if it's because Mellanox has made their own tweaks. >>=20 >> I would hope that the very latest OFED from Mellanox would include an=20 >> up-to-date and working set of libs/modules, but maybe you're on to=20 >> something. It sounds like a risky move, but maybe I'll try just=20 >> installing rdma-core from github over top of the OFED install. I=20 >> have a fear that I'll end up with inconsistent versions, but it's worth = a try. >>=20 >> Thanks, >> --Jim =20