From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4D8F1A0C41 for ; Thu, 18 Nov 2021 20:56:58 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CDBEC40687; Thu, 18 Nov 2021 20:56:57 +0100 (CET) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-oln040092075025.outbound.protection.outlook.com [40.92.75.25]) by mails.dpdk.org (Postfix) with ESMTP id 53D9440395 for ; Thu, 18 Nov 2021 20:56:57 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iY8BTuJPz1Jc7bYGPqU5+Tpa0PIX3aa2wAfUX7HO+2XxHJsDh4W4M05VNdmNRqy/EXTqd/xT/EVDgRBlc6koAlpYjdQIpnHlB8WENxXUqVDEtNPxMrQJhg6vczKVPLHbwEc1DVokpmz8znqiOym5bO6aM7QOu5O29eSz0GlP4QWxZQshClaVRup0z4GjnpIB/Xxp9BuDT4P9QU5V27prDRx30vQ1DbAEDfUmdQ9EjpoT5VBqqCSVjcLWMvMbKJuIYaZVeietsfoPMerNRR3osgKyfqDPxS6dR1UtyqLgl3cFMPKPgBLrdN0uBsnVU1Hxq+pHr6O9nhGsvmMga/jggA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JPNp2sKGqUNWh3ETg1/QA8XomrncBgAP36UAmFVr+/E=; b=KXamwbLo/Gqo+BJWRlOQnkX5SJgRb8PpK1aV5dnfK1X1ps639612trl3bWarIVCVMY38AGfUitYdfCXKvtCjvFEgfJkhFRglyFbu75HjWkc6/EJKYmtj/RppLF1BE7WgpYouNr48+k1GhROQ3RGmhFR6Um3zSHXBiplZ0n/BS8o9+R0a0hqDaYxFlpWYS2Culpu0zEuWzs5eo2gHKRK5iDNH0B0TA6qjMTd3DbzCQ0tmFiZguiTPVINYY0QSRBql99p7n459AQwEWgF5MekeOYgwZUUy0CYKa4JbM8uCJKlimRr+BJgxFhjuLZYdNLRvsaxOg8zDxsJS/3Ay1Z2iHw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JPNp2sKGqUNWh3ETg1/QA8XomrncBgAP36UAmFVr+/E=; b=FZ8gfK/Y3jsG9amvgDFdziVZZjD1odfanNdtjbLQJVeujxOmzozgOb2Sc81twV/wN0Ong87uxDk4D3UwbMwjImuhZzSlmyvqMG91sZmuYoI7VP2awn5wTpZTNRKMBXJdPq2LjaBcKpDlkDPUtTWydE2N6Z7ykULIlRLw/b+nIorUnxng4WJnMe+sHfdBcPwi/vErU2oW0Rla04ef2sQ9FQBJWFFgUqacrooy9zII4y5YwXN4HVl4nOrOjBSOfqD5O5+PGiqOxmXTmpkg3JNyNGp9vDfO3clc31+ZikEoKbdLpqX08787OIw4C3+R5HszjiUi33oP+qTpMkcY/Rt8Ug== Received: from AM6P192MB0437.EURP192.PROD.OUTLOOK.COM (2603:10a6:209:2e::11) by AM7P192MB0770.EURP192.PROD.OUTLOOK.COM (2603:10a6:20b:14d::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4713.19; Thu, 18 Nov 2021 19:56:56 +0000 Received: from AM6P192MB0437.EURP192.PROD.OUTLOOK.COM ([fe80::482f:973c:47be:6706]) by AM6P192MB0437.EURP192.PROD.OUTLOOK.COM ([fe80::482f:973c:47be:6706%3]) with mapi id 15.20.4690.027; Thu, 18 Nov 2021 19:56:56 +0000 Date: Thu, 18 Nov 2021 19:56:55 +0000 (UTC) From: David Schramm To: users@dpdk.org Message-ID: Subject: KNI and mbuf pool with external memory Content-Type: multipart/mixed; boundary="----=_Part_19_165048050.1637265415716" X-Correlation-ID: <19c6ef6d-4997-4c03-a2db-be7d28566e56@live.com> X-TMN: [UrMn+6DRrVi5Q5HWySGGyzfRiP4Jq4CV] X-ClientProxiedBy: AM5PR1001CA0015.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:206:2::28) To AM6P192MB0437.EURP192.PROD.OUTLOOK.COM (2603:10a6:209:2e::11) X-Microsoft-Original-Message-ID: <19c6ef6d-4997-4c03-a2db-be7d28566e56@live.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [127.0.0.1] (87.159.50.136) by AM5PR1001CA0015.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:206:2::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4713.19 via Frontend Transport; Thu, 18 Nov 2021 19:56:55 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d85037dc-c22c-4241-e1ca-08d9aacd92fc X-MS-TrafficTypeDiagnostic: AM7P192MB0770: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FmKHVmFXU6mVl1rKDlII+aY5lAYBSCDlH6T85Zp9V5Bp+YS3ToZW4DyeG5wHOWNNXXqZqnQz8Q4rpb9xzmsGcvHfxIZrm7bnxG7m+xQwq/4X4v9b60BHTUmn9Tql8VJNGpAe/befNTf1nUz77IJ4kqkAvOGiSuPZ2hzUupqd7WQMC1qdg48ZiUf5kI/LBz2joq7MTZjr1NSKBANo39Awc7oa1rRLoWOCIZ8lLbA1u4uAxNspMNJeJ1gC390L51fwI81s8cQ0dT+7++ximmaxzBZHfeEAlfCFMPRvcXKFA+HhnA75lxqItSgobNE2Ka9mHjDFAW1Ky77Mb9Ff81NDMzBxpk0GL9y4lPoqC3IZO3sAnDulMV8l/aYTpsppRoAm9aUOfNxvg2dVrR4+brFHEAa6FjiD7jxhbf2ZwjV9s6+UGDR+uCHVWbI2ewGkNDiVQ8H7A4ZXvWrpshTRG+l0pGc0eF+K9tAQYAItGt9GJeuzR0I8W6JC/1ELtKrb6hdSvUiHFPFg0icYopl1814wPQG2ZNsOgEpbeJSb2AAaKa92ma1sPs4dbHFXRArsqrKlXq0yjcYuh5YIW6g2LFA3d1HYkylF5d+xKQnJ6l5iSIJX49bu+CVxra/Jn6aUJb1o X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: RMP2FCfM5/v8cClF5A5Q3bE93sHVzarOI11CplRcFA7yPnw9+55oQO7tkjtHu8doS9+bVkPfjlrn0+Es/bE2ew4pvJhqUUc0EMuNujPn4/2slKEBhe/gn4zPoMwPW9znhDz5ktJkFNY2DuwTZ2joVqJThTkXhTSYutwZ1WQZITls3m9EA8B45V2ahxN3F2Dxk+0bMWwRAsi0Ck7vOjY8G6SJSWZlXv9faJnbcwag6vzOW46orpu9OGMpdg8sWxEFToIo5EuzqjTKo6qik2Kl4/S57HW9yDbzqjZAkJY3kf7iy33An3ZcLugcrEJ5On8r0Jvj4coSyKPhy3r9JcvxKkgO55PS0mBRapoxJJVCT19WlpMVNH0fMOYXfYWC53UgLV6w2E3E1loHCLiOq7Gse43wakfFriCKhBdPcuvIAIqpJtKyklDHCQMvpJ/mWqjEOU5q2LMIHl4IKLV+cmRQsPK+pU4CZa9ugW7QARRV2xy4ptFP+/6yICkPJQa11V9UWLKFsyuYDx2+a3tEoPYCto20Oi135OCeoWrt1oSgj/7c+zjLkqsutus7xlX+j7IX6lVTiEv7wJqOoc/5Zxde464rCWtYXkGiT170rFveQrA0QzheiKUsNhwVFaIwO4XoFSidiBANO68HIWbhOiZbejT3Oqrsc2Q1vq2HyTtMUtv1HoxX4UL7TgGQgYpQQv8xe8WDqHoZbJek6BIQ0u4f/w== X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-729fd.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: d85037dc-c22c-4241-e1ca-08d9aacd92fc X-MS-Exchange-CrossTenant-AuthSource: AM6P192MB0437.EURP192.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Nov 2021 19:56:56.3588 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7P192MB0770 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org ------=_Part_19_165048050.1637265415716 Content-Type: multipart/alternative; boundary="----=_Part_18_248222205.1637265415716" ------=_Part_18_248222205.1637265415716 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, is it possible to use the kni with a mbuf pool which uses external memory? In my attempt, the kni module seems to crash when it tries to access a mbuf= from the pool: [Do Nov 18 18:56:08 2021] device fw0 entered promiscuous mode [Do Nov 18 19:03:38 2021] rte_kni: Creating kni... [Do Nov 18 19:03:53 2021] general protection fault, probably for non-canoni= cal address 0xae88ba513f6f816e: 0000 [#1] SMP NOPTI [Do Nov 18 19:03:53 2021] CPU: 4 PID: 0 Comm: swapper/4 Tainted: P=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 OE=C2=A0=C2=A0=C2=A0=C2= =A0 5.14.0-2-amd64 #1=C2=A0 Debian 5.14.9-2 [Do Nov 18 19:03:53 2021] Hardware name: Gigabyte Technology Co., Ltd. B560= HD3/B560 HD3, BIOS F4 03/16/2021 [Do Nov 18 19:03:53 2021] RIP: 0010:memcpy_erms+0x6/0x10 ... ... [Do Nov 18 19:03:53 2021]=C2=A0 kni_net_tx+0x163/0x3a0 [rte_kni] [Do Nov 18 19:03:53 2021]=C2=A0 dev_hard_start_xmit+0xc1/0x1e0 [Do Nov 18 19:03:53 2021]=C2=A0 sch_direct_xmit+0x9d/0x370 The full log is also attached to this mail. I'm trying to forward packets b= etween a NIC and a kni interface, when using rte_pktmbuf_pool_create, it wo= rks as expected. The corresponding code for creating the pool is here: https://github.com/da= schr/cuda_firewall/blob/6b20d6e45f558085157d0c279b077a9610b5aa35/misc.c#L61 The whole code: https://github.com/daschr/cuda_firewall/tree/plain-fwd-kni Another question: Do benchmarks of the kni exist? What are the common line/= packet rates one can expect when using the kni module? In my tests, I curre= ntly get ~20Gbit/s using iperf3. Cheers, David ------=_Part_18_248222205.1637265415716 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit Hi,

is it possible to use the kni with a mbuf pool which uses external memory?

In my attempt, the kni module seems to crash when it tries to access a mbuf from the pool:

[Do Nov 18 18:56:08 2021] device fw0 entered promiscuous mode
[Do Nov 18 19:03:38 2021] rte_kni: Creating kni...
[Do Nov 18 19:03:53 2021] general protection fault, probably for non-canonical address 0xae88ba513f6f816e: 0000 [#1] SMP NOPTI
[Do Nov 18 19:03:53 2021] CPU: 4 PID: 0 Comm: swapper/4 Tainted: P           OE     5.14.0-2-amd64 #1  Debian 5.14.9-2
[Do Nov 18 19:03:53 2021] Hardware name: Gigabyte Technology Co., Ltd. B560 HD3/B560 HD3, BIOS F4 03/16/2021
[Do Nov 18 19:03:53 2021] RIP: 0010:memcpy_erms+0x6/0x10
...
...
[Do Nov 18 19:03:53 2021]  kni_net_tx+0x163/0x3a0 [rte_kni]
[Do Nov 18 19:03:53 2021]  dev_hard_start_xmit+0xc1/0x1e0
[Do Nov 18 19:03:53 2021]  sch_direct_xmit+0x9d/0x370

The full log is also attached to this mail. I'm trying to forward packets between a NIC and a kni interface, when using rte_pktmbuf_pool_create, it works as expected.
The corresponding code for creating the pool is here: https://github.com/daschr/cuda_firewall/blob/6b20d6e45f558085157d0c279b077a9610b5aa35/misc.c#L61
The whole code: https://github.com/daschr/cuda_firewall/tree/plain-fwd-kni

Another question: Do benchmarks of the kni exist? What are the common line/packet rates one can expect when using the kni module? In my tests, I currently get ~20Gbit/s using iperf3.

Cheers,

David
------=_Part_18_248222205.1637265415716-- ------=_Part_19_165048050.1637265415716 Content-Type: text/plain; charset=UTF-8; name=dump.txt Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=dump.txt [Do Nov 18 18:56:08 2021] device fw0 entered promiscuous mode [Do Nov 18 19:03:38 2021] rte_kni: Creating kni... [Do Nov 18 19:03:53 2021] general protection fault, probably for non-canoni= cal address 0xae88ba513f6f816e: 0000 [#1] SMP NOPTI [Do Nov 18 19:03:53 2021] CPU: 4 PID: 0 Comm: swapper/4 Tainted: P = OE 5.14.0-2-amd64 #1 Debian 5.14.9-2 [Do Nov 18 19:03:53 2021] Hardware name: Gigabyte Technology Co., Ltd. B560= HD3/B560 HD3, BIOS F4 03/16/2021 [Do Nov 18 19:03:53 2021] RIP: 0010:memcpy_erms+0x6/0x10 [Do Nov 18 19:03:53 2021] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d= 1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 4= 8 89 d1 a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe [Do Nov 18 19:03:53 2021] RSP: 0018:ffffa60640234cd0 EFLAGS: 00010213 [Do Nov 18 19:03:53 2021] RAX: ae88ba513f6f816e RBX: ffff98eee0442000 RCX: = 0000000000000036 [Do Nov 18 19:03:53 2021] RDX: 0000000000000036 RSI: ffff98ee89cbb002 RDI: = ae88ba513f6f816e [Do Nov 18 19:03:53 2021] RBP: ffff98eea0b75400 R08: 0000000000000000 R09: = 0000000000000001 [Do Nov 18 19:03:53 2021] R10: ffff98eee0325300 R11: 0000000000000005 R12: = 1182dd106338dd32 [Do Nov 18 19:03:53 2021] R13: 0000000000000036 R14: ffff98ed8243fd82 R15: = 0000000000000036 [Do Nov 18 19:03:53 2021] FS: 0000000000000000(0000) GS:ffff98f22fb00000(0= 000) knlGS:0000000000000000 [Do Nov 18 19:03:53 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Do Nov 18 19:03:53 2021] CR2: 00007f82819dfc10 CR3: 00000002bc210001 CR4: = 0000000000770ee0 [Do Nov 18 19:03:53 2021] PKRU: 55555554 [Do Nov 18 19:03:53 2021] Call Trace: [Do Nov 18 19:03:53 2021] [Do Nov 18 19:03:53 2021] kni_net_tx+0x163/0x3a0 [rte_kni] [Do Nov 18 19:03:53 2021] dev_hard_start_xmit+0xc1/0x1e0 [Do Nov 18 19:03:53 2021] sch_direct_xmit+0x9d/0x370 [Do Nov 18 19:03:53 2021] __dev_queue_xmit+0x5aa/0xb00 [Do Nov 18 19:03:53 2021] ip_finish_output2+0x194/0x590 [Do Nov 18 19:03:53 2021] igmp_ifc_timer_expire+0x1d7/0x420 [Do Nov 18 19:03:53 2021] ? igmp_start_timer+0x70/0x70 [Do Nov 18 19:03:53 2021] call_timer_fn+0x24/0xf0 [Do Nov 18 19:03:53 2021] __run_timers.part.0+0x1ca/0x230 [Do Nov 18 19:03:53 2021] ? cpumask_next_wrap+0x2d/0x70 [Do Nov 18 19:03:53 2021] ? _nohz_idle_balance.constprop.0.isra.0+0xb6/0x2= 70 [Do Nov 18 19:03:53 2021] run_timer_softirq+0x26/0x50 [Do Nov 18 19:03:53 2021] __do_softirq+0xc2/0x268 [Do Nov 18 19:03:53 2021] irq_exit_rcu+0x99/0xb0 [Do Nov 18 19:03:53 2021] sysvec_call_function_single+0x72/0x90 [Do Nov 18 19:03:53 2021] [Do Nov 18 19:03:53 2021] asm_sysvec_call_function_single+0x12/0x20 [Do Nov 18 19:03:53 2021] RIP: 0010:cpuidle_enter_state+0xc7/0x350 [Do Nov 18 19:03:53 2021] Code: 8b 3d 65 49 35 62 e8 e8 b4 a0 ff 49 89 c5 0= f 1f 44 00 00 31 ff e8 69 bf a0 ff 45 84 ff 0f 85 fa 00 00 00 fb 66 0f 1f 4= 4 00 00 <45> 85 f6 0f 88 06 01 00 00 49 63 c6 4c 2b 2c 24 48 8d 14 40 48 8d [Do Nov 18 19:03:53 2021] RSP: 0018:ffffa60640143ea8 EFLAGS: 00000246 [Do Nov 18 19:03:53 2021] RAX: ffff98f22fb2ca80 RBX: 0000000000000003 RCX: = 000000000000001f [Do Nov 18 19:03:53 2021] RDX: 0000000000000000 RSI: 0000000020b84733 RDI: = 0000000000000000 [Do Nov 18 19:03:53 2021] RBP: ffff98f22fb37000 R08: 000019a3ca7f9d9f R09: = 0000000000000018 [Do Nov 18 19:03:53 2021] R10: 000000000000085e R11: 0000000000000c50 R12: = ffffffff9edb9960 [Do Nov 18 19:03:53 2021] R13: 000019a3ca7f9d9f R14: 0000000000000003 R15: = 0000000000000000 [Do Nov 18 19:03:53 2021] ? cpuidle_enter_state+0xb7/0x350 [Do Nov 18 19:03:53 2021] cpuidle_enter+0x29/0x40 [Do Nov 18 19:03:53 2021] do_idle+0x1f6/0x2b0 [Do Nov 18 19:03:53 2021] cpu_startup_entry+0x19/0x20 [Do Nov 18 19:03:53 2021] secondary_startup_64_no_verify+0xb0/0xbb [Do Nov 18 19:03:53 2021] Modules linked in: sd_mod uas usb_storage nvidia_= uvm(POE) sch_ingress sch_multiq tun rte_kni(OE) rfcomm intel_rapl_msr intel= _rapl_common bnep mei_hdcp x86_pkg_temp_thermal intel_powerclamp coretemp k= vm_intel nvidia_drm(POE) kvm drm_kms_helper irqbypass cec ghash_clmulni_int= el rc_core aesni_intel btusb btrtl btbcm snd_sof_pci_intel_tgl btintel cryp= to_simd nvidia_modeset(POE) cryptd snd_sof_intel_hda_common intel_cstate bl= uetooth soundwire_intel soundwire_generic_allocation soundwire_cadence snd_= sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_h= da_ext_core snd_soc_acpi_intel_match snd_hda_codec_realtek snd_soc_acpi snd= _hda_codec_generic snd_soc_core snd_compress snd_hda_codec_hdmi soundwire_b= us binfmt_misc intel_uncore ledtrig_audio snd_hda_intel snd_intel_dspcfg sn= d_intel_sdw_acpi jitterentropy_rng pcspkr nls_ascii snd_usb_audio snd_hda_c= odec nls_cp437 iTCO_wdt snd_usbmidi_lib efi_pstore intel_pmc_bxt vfat iTCO_= vendor_support sha512_ssse3 snd_rawmidi [Do Nov 18 19:03:53 2021] nvidia(POE) intel_wmi_thunderbolt wmi_bmof fat w= atchdog ee1004 sha512_generic snd_hda_core snd_seq_device ctr mc snd_hwdep = drbg snd_pcm ansi_cprng joydev snd_timer snd ecdh_generic mei_me rfkill ecc= soundcore mei sg evdev intel_pmc_core acpi_pad acpi_tad sunrpc parport_pc = drm ppdev lp parport fuse configfs efivarfs ip_tables x_tables autofs4 ext4= crc16 mbcache jbd2 crc32c_generic mlx4_ib ib_uverbs ib_core mlx4_en ptp pp= s_core hid_generic usbhid hid sr_mod cdrom xhci_pci ahci xhci_hcd libahci l= ibata mlx4_core usbcore r8169 nvme realtek scsi_mod mdio_devres nvme_core l= ibphy crc32_pclmul crc32c_intel i2c_i801 i2c_smbus t10_pi crc_t10dif crct10= dif_generic crct10dif_pclmul usb_common crct10dif_common fan wmi video butt= on [Do Nov 18 19:03:53 2021] ---[ end trace c5cf0670bde49bd2 ]--- ------=_Part_19_165048050.1637265415716--