From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7CE0EA034F; Wed, 10 Nov 2021 00:34:30 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6335440683; Wed, 10 Nov 2021 00:34:30 +0100 (CET) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mails.dpdk.org (Postfix) with ESMTP id 26C1B40142; Wed, 10 Nov 2021 00:34:29 +0100 (CET) Received: by mail-wm1-f48.google.com with SMTP id 71so569415wma.4; Tue, 09 Nov 2021 15:34:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=7xVwLomigo2RIxBXBympgWwLiDGv68KqGCByZguZ2eM=; b=ckj5WqoRy5OHOvOjW0Hesqzdx4hVg6RpKn11+bojd/q8Lf0w6aa16bybTLnS1KMieh 7WLYOjW5O719Jbur496hXSX22c9Q95sXjvwkG1M2y5GV3t8CJ9m5UYJXkl+cajiCcVxy X7+d5TF3/U3T3XTeCBOfarKHg3a+uA4cKgzMgkDSACpI+wZTL1SXjs17ryHG/YM+SQ/F YGRne7mzZ/DWEIcgPaeJ8MWjiXVwfeIVDYOGXJkouXMu3ov+rwi772AzlvVeseFDemRV MAWTLLNMvTJjZVBfvpz3k/mD2RB+BQoQc9kV7lhl3/DMxX4fYXq1uKgkz8R90FK19Z3b T8MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=7xVwLomigo2RIxBXBympgWwLiDGv68KqGCByZguZ2eM=; b=jgRwL7370mKzQqwKcfec0AswYI0nS0vNuGAv9TTwxxhkuGB6uvjfmAOln+4g4uJWfM 9VzKSwbmPREZYNu6fYLBF86PfkTvS3uQhSgSwzAaXX2meZ6KJPv/Qs0tFaz5cPh+Xa0Y XY9am+RO0bKIGnnc3Mk/kjAJihCIOXgBP5KLjYDGZ+43QVRGhoUlKXnla9LhI1WWfc1n xp7ggzMFJDH3kQz0+wkj+VND0994bJm6yjDWjyDwBR+96xIawesXsRjlK2ZiwQRtmpvn YKXqzn9ZyhbWLYeT4ftZy2XnipwnQOcVs/4KFT3HcSirobR185nRCuqbNcnRkRyPTKL6 nZ3g== X-Gm-Message-State: AOAM5317Jf2XvQ1atFFVyORQEEYVf/RGt/+FK9x2eFOBq4QW2K9gmVu8 7nQ/NXoJFh6jeDNYTcOAWpUy9NuTAymPDJ69cRF1WaLSmVA= X-Google-Smtp-Source: ABdhPJzpLfYAcB/cABfbTkyW34ixfWcP2QvVx1hYo7qCi+8WwjvidIvZ3d0v1E/AprnjoLroJVz+8a7SEELrofcJZzQ= X-Received: by 2002:a05:600c:4982:: with SMTP id h2mr11683195wmp.4.1636500868510; Tue, 09 Nov 2021 15:34:28 -0800 (PST) MIME-Version: 1.0 From: fwefew 4t4tg <7532yahoo@gmail.com> Date: Tue, 9 Nov 2021 18:34:17 -0500 Message-ID: To: users@dpdk.org, dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [dpdk-dev] RSS Hash suport in DPDK on Amazon ENA NICs broken? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On AWS EC2 I configure: - c5n.metal instance - Stock AWS AMI Ubuntu 20.04 LTS ($ cat /proc/version Linux version 5.4.0-1045-aws) - Apply the vfio pci driver from https://github.com/amzn/am zn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch - Using DPDK code from head of main branch. I built using ninja/meson - Add a 2nd ENA enabled NIC - Setup 2Mb huge pages, and bound the 2nd NIC with dpdk-devbind I run test-pmd with this command: $ sudo ./dpdk-testpmd --huge-dir=/mnt/huge -c 0xf -n 4 -- -i EAL: Detected CPU lcores: 72 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 1048576 kB hugepages reported EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Ignore mapping IO port bar(3) EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0) TELEMETRY: No legacy callbacks, legacy socket not created Interactive-mode selected Error picking flow transfer proxy for port 0: Function not implemented - ignore testpmd: create a new mbuf pool : n=171456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 16:9A:11:4A:5D:61 Checking link statuses... Done Error during enabling promiscuous mode for port 0: Operation not supported - ignore *Key point: I try to enable RSS hash offload for UDP or TCP. I get the following error shown in bold:* testpmd> port config all rss udp *Port 0 modified RSS hash function based on hardware support,requested:0x20820 configured:0x820*ena_rss_hash_set(): Setting RSS hash fields is not supported ena_rss_hash_update(): Failed to set RSS hash *Configuration of RSS hash at ethernet port 0 failed with error (95): Operation not supported.* testpmd> port config all rss tcp *Port 0 modified RSS hash function based on hardware support,requested:0x10410 configured:0x410*ena_rss_hash_set(): Setting RSS hash fields is not supported ena_rss_hash_update(): Failed to set RSS hash *Configuration of RSS hash at ethernet port 0 failed with error (95): Operation not supported.* If I run ethtool on the NIC before putting it down state and DPDK binding to vfio-pci I see an indication that there is a hash key, and RSS Hash works: ubuntu$ ethtool -x ens6 RX flow hash indirection table for ens6 with 32 RX ring(s): 0: 0 1 2 3 4 5 6 7 8: 8 9 10 11 12 13 14 15 16: 16 17 18 19 20 21 22 23 24: 24 25 26 27 28 29 30 31 32: 0 1 2 3 4 5 6 7 40: 8 9 10 11 12 13 14 15 48: 16 17 18 19 20 21 22 23 56: 24 25 26 27 28 29 30 31 64: 0 1 2 3 4 5 6 7 72: 8 9 10 11 12 13 14 15 80: 16 17 18 19 20 21 22 23 88: 24 25 26 27 28 29 30 31 96: 0 1 2 3 4 5 6 7 104: 8 9 10 11 12 13 14 15 112: 16 17 18 19 20 21 22 23 120: 24 25 26 27 28 29 30 31 RSS hash key: 80:d3:0f:c4:1d:8a:48:9e:7f:1d:18:91:d3:12:cb:38:87:fc:33:49:18:30:e4:1d:19:8f:9d:fd:c3:94:fe:f3:b2:f0:7f:e9:e9:9d:2d:af RSS hash function: toeplitz: on xor: off crc32: off What gives? Do I need to modify a constant somewhere in DPDK? At the risk of expanding scope, I am really trying to run eRPC which runs on top of DPDK. It fails in dpdk_init with a similar but slightly different error shown in bold in which the putative valid value is different than above: $ sudo ./hello_server 172.31.77.76:31850 20:536566 INFOR: eRPC Nexus: Launching 0 background threads. 20:536582 INFOR: eRPC Nexus: Launching session management thread on core 53. 20:536776 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 172.31.77.76. 20:536832 INFOR: DPDK transport for Rpc 0 initializing DPDK EAL. EAL: Detected CPU lcores: 72 EAL: Detected NUMA nodes: 2 EAL: Auto-detected process type: PRIMARY EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 1048576 kB hugepages reported EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Ignore mapping IO port bar(3) EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0) TELEMETRY: No legacy callbacks, legacy socket not created 21:428883 WARNG: Running as primary DPDK process. eRPC DPDK daemon is not running. 21:428893 INFOR: DPDK transport for Rpc 0 got QP 0 21:428927 INFOR: Initializing port 0 with driver net_ena *Ethdev port_id=0 invalid rss_hf: 0x20820, valid value: 0xc30*terminate called after throwing an instance of 'std::runtime_error' what(): Ethdev configuration error: Invalid argument Aborted