From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0062.outbound.protection.outlook.com [104.47.2.62]) by dpdk.org (Postfix) with ESMTP id 0B10C7CC2 for ; Sun, 27 Aug 2017 08:47:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=BTgiLILT/99zfYIqC9nGiqHVWkawLl4OH4Fxevh+Du4=; b=shywYVxnBUmzUfpbzGYtWId6ps6+XWQLiwuzRGI12dJdQR6qnSIfmIBO+li9d74pbdTDpHBCDI0/z2jLmszDn9ei9nEwP6V3jZG9rp7hzAOYzANv6pi1km5shFGolqKKGcqxCi8RboWlzhjsVRIhr1fa3Q9r3Q/yoeUnRlHoY6M= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; Received: from mellanox.com (82.166.227.17) by AM4PR05MB3137.eurprd05.prod.outlook.com (2603:10a6:205:3::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1385.9; Sun, 27 Aug 2017 06:47:14 +0000 From: Shahaf Shuler To: nelio.laranjeiro@6wind.com, adrien.mazarguil@6wind.com Cc: dev@dpdk.org Date: Sun, 27 Aug 2017 09:47:07 +0300 Message-Id: X-Mailer: git-send-email 2.12.0 In-Reply-To: <1503301622-14220-1-git-send-email-sagi@grimberg.me> References: <1503301622-14220-1-git-send-email-sagi@grimberg.me> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [82.166.227.17] X-ClientProxiedBy: AM5PR0502CA0023.eurprd05.prod.outlook.com (2603:10a6:203:91::33) To AM4PR05MB3137.eurprd05.prod.outlook.com (2603:10a6:205:3::14) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2edc2d6d-751e-4980-a7f7-08d4ed17738e X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(48565401081)(300000503095)(300135400095)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:AM4PR05MB3137; X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB3137; 3:+dWI90s8NpfS9FxMpHyYepTZM1vUUvuuqVD38F6t320lJHLfnT9vFED/OH5tl1RQia7Y6+Rwt2aoCvR8mhb5G6HVweFqyw+byAw+m9ybLTekZJ5bhaqEPF0KX6Qt89qrdmnjlAe56ynSkNrEOIZUu31fi7Xn0T/7aB51o6om5vbHnp/RioJMxPp1R4IX7w1ZX6TSaZBqSJ36VcIByeZMmpz14fwzJ+cmlOFjzetljAY0IIQCpKlJxdmsAsTkLkH9; 25:ECtnLBjtr7mhVLGv8nyQEyfNuS4acqrT3GRDiaM1oavlhJT2cESZSnJ+0DZykiU/PjINgWHY/UMyQFAuPkr3qsoNNWcE6TKjdyRYOe4f4Q5EO8Wx1iTBWrgEgl0vp2gz+KuIUKOaw5Q3cwfzUwY/HU7tCeZw7np5sEk/cNQx4VDC3qJSCoWZHpG1D2BkB+PnqUOGOPqgZE20HmJTkghQYDkhR6FSgKM+89AU6u2Fq1FBgaTq0SSMEAa8MC/g7zYyN1qUj0x0CyZ2sBSR4fxuMHDi5zpsU48EHCF+t+CKsbg6bhci2nvDbQtNEvxUPuMjN5cIBXMj/jO3dhxD2X+ZUg==; 31:iLhDZeKORSLh6EMmx5gwKox0Ls2G/rijmQgbVfPab+yOqootnC6H8rK671X8XiZ5MZJyv5HM6GpPj/eCLUHsgHUtrWkD6GszPFsHBu5YfjQ1SpXswQiphn4O/AoSq7ndmhVm4cUHP6Kx2WSHsaXTkYOHDGxKdRwlregwAYZVko2Qs5sNQEKRL5G69baM9tTyjJIvcunU7RqeEMKcaX8Omf3tI2NozqLy1pfi2GTeYyw= X-MS-TrafficTypeDiagnostic: AM4PR05MB3137: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB3137; 20:aakap405pUsAgQbyZmm5C+lYYCVgXeP4tuj4gDqxoI/PngkLXsH+sTj9a5NdUx3PqS3snDCSdx2uQTA0Hvc+HLFhYPX1ytwA5lV1osFBoDOYTv4JNUdgzCYureSw96iReXRousjhDU8e73lDO8W+0NsCvz1FPTrRx2sYzefWkcLluyv5oTkFBoyF7fQrRWMxYXdFWg4f2JD8sTE0m6xQlC4nGTReMcGsjHpL9xDW5aB5h8t7K04VlkRKbtrb+9BDX0J2zji/lfsiHEOCDSvIVskOLhceNk82JLlrFNKbv8kOS4tVr/szM6liIRYv4p9HcofthsSQ0XMsOIzh4quv7S+Nj8ECyTM8lbSOWzJR+sQ7dxwzseMDy+Vrch7GqwJuettUSNb+YHUsAWi+rdPJ+UiqDjThPlQghbTVRnMX1tyWKREiUroQ6fAHz1np1LvLHUsa5onartW8hbT+DvMA5zw43vwa3j4pQ1eO5IHP4BiyBR40O6di45OXs3wgJN/k; 4:LHFSi/sKP10u6KV/BK93Ypa+vT6NGcj7VrTx+MdKb7rBY3HPXdBrTIdbYBDsWyPYvTFZVjpOH0A1hePZEFoY2A/bxMGaw7I2Eorkd0CB7CpcOpmgyTaXnQrvPXxkvDnSa8A74y8/2rX+JC5GAwvZ6cRRxq+0F3kiBSahm2nRGpcGpPvNmKsRPOEf/X1aNgHpTEThaj6D8+NPS6LI2WxKoBvpi23hVZTCVoqjb3E6X/SWc7xH2kuhXKhSjskE056AnpTq111xuIY+FDMp3kTuGrDStlKGpuV1Gmid3D7rg0Y= X-Exchange-Antispam-Report-Test: UriScan:(788757137089); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(10201501046)(93006095)(93001095)(3002001)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:AM4PR05MB3137; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:AM4PR05MB3137; X-Forefront-PRVS: 0412A98A59 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(7370300001)(4630300001)(6009001)(39860400002)(189002)(199003)(33646002)(53936002)(21086003)(68736007)(25786009)(5003940100001)(189998001)(105586002)(97736004)(101416001)(33026002)(6116002)(4326008)(3846002)(55016002)(478600001)(106356001)(110136004)(36756003)(48376002)(305945005)(50466002)(2906002)(6666003)(76176999)(2950100002)(50986999)(42186005)(8676002)(69596002)(50226002)(81166006)(5660300001)(47776003)(7350300001)(66066001)(86362001)(7736002)(81156014); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR05MB3137; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM4PR05MB3137; 23:Lo/iL46HIupDUnFvSWRiZi2+BUrPYr02Ule+UWTFH?= =?us-ascii?Q?ymSebgrytfuQn1pE5wN5a+XKsERhfhcO6HwKSoIjzy5ciplE/33uaYs9xvJn?= =?us-ascii?Q?hZ/OOBJjlBfqnI2sfLNFFIhExfu6XhRWI7pyTJ3LWPFlJpgojgk2AmsArO3v?= =?us-ascii?Q?2LpD/YhC98GAyRKjtg+N7VUlErrWleQo+9W6uLvlTkULFxDKa1Wx2X4nQ4tJ?= =?us-ascii?Q?kd21sAJ4aNwdDKiMMSpeghErA6jgyhsqE9+nRxS1jXEriAIvoVLMuU0Dx0VT?= =?us-ascii?Q?OIWIonJTNEiM4EnBI1BxO5FICkB/fpGS2Y7Ni/yJHCPTdySmfkSHD+ox906D?= =?us-ascii?Q?AYBYwtCDWRYuX+g+JvzvtLVKVTBIdlDSjZU3t6dISEvpkR14yT1ufQiobGRQ?= =?us-ascii?Q?0zIGgkWmZN1JulrCEM5ohDfkrLPDrYBCJUvDkzQKiu/Rv6vNV7G8AXo/OR7N?= =?us-ascii?Q?2IPEH0TN6gxX0p8lsroKZ2BAD2Da8gGnyAdaZPC2Ad9thhZR/U4nOPvuzfpL?= =?us-ascii?Q?xqgs8cmIJq9wHzXsdVStreDZktqUaArJofp5MTWR2hu+QYjd9FKiz7tXkZKg?= =?us-ascii?Q?Vvya2fggQgYF6JV1aVBYXrUObg3Sm53dAn3d9bn50ybIvaRWxKTgCs161043?= =?us-ascii?Q?VbW+FoJ639c8XqnDX7HbYpS2cmdatp0cnfxdtXmeu4w2zgIhpzHDWLVIo8/1?= =?us-ascii?Q?ApNFpdQkBChYE6DlFowdThMW1XDV77UVcVxFx4ull0jE5HVEqS6p2/ZFs1T2?= =?us-ascii?Q?MWXV+s2LsC4CB3MNBVVbql4oBEywx6sCm7Fh/ePFvN4Ftx97P08HJKdJLxzf?= =?us-ascii?Q?FXVOVMRQZ7sM99ILvp3wr0Ih/8bjqzlJ9ssW1ZY/VKdf9pOeQ2JZPRKaLomw?= =?us-ascii?Q?tXhog2z1VOg7c7pVa4AC/pxUrRSVD4gxupStfY/cc5+AWxRVK0mLNVUSxseq?= =?us-ascii?Q?dbQ+2EZmuwC478NPYyt6gPkyqpD+RYhZ/1yRza+TTlMN+dirQHGkVZJ9ojMf?= =?us-ascii?Q?7NLvKpEPeK7/dH0EYUmq84sBfEx5oLpw5xWJVvwdGNeXIelhrP4zaMP5WVYx?= =?us-ascii?Q?uIc6D6m9pOCubMLZaKQrmilzrcuFfybgum5mJJJU88OITzinolTFaukeWYzN?= =?us-ascii?Q?Lu0TjUuc/4=3D?= X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB3137; 6:51OzGmiVnZodcVxL1r9/L5h03AWCOJIZlMrDSBs6uaRivPudB+oMACjoAZmPyKiJlDxzQGbDlJwb26+IP+Tl/FuYqMM5KJGyQ22m7qJfUwmH9CIFI5Vw+bLSFwizwxynyokokIVR9WcQdt0qdi2LVd2DK7KBGt4ittjbiiRtFK7s4IQTEHmO0xQPs8Bdn00klRJM0oiwos7vEnMEgonT0+JlkTgXB04B0ORcUrQI/8F0EmTTjjSqiMa3g5v81XCJWjAf5xB847KpgeVTZBDVffl32XvuJQKSoe1VyiYwdd3SBcifCwvBCCKxkQd14BdGwlWsM00ZPmStPhb7OOWwrA==; 5:UHVYcVvNOya2mVyEclCla/L50zR+uVIbzzMHlDLa0d/O+0yd0nN31be1HLI/2Q4VSrx5W3zZ+P7KXF7Q3NK45khDyRc16M5izkjxEpbmqoM9xxSfudkOeLji1sz8miv56Oz5TaQkNQ/s2BzE4NcoPA==; 24:lQF2U8t+y65gXSt2IVfLLbmeLGs82HkbmndwDMJkmO+KcHR3SuuQS6sVUc7evP9PP5vrea5pZ56L3y+AJIz65h+eSf5zA64g0ll1eGOdXg0=; 7:SD9D4nalBQMW3eG+NM6TLIORhem7tTOE9b1t8muPGPGb6uIeqoy61XjU/IVx0swGgRmcnMXGwtVKFxXlDvtHazyOy7mN5cX0Lfww/LW4rjEPxDVHDVFv9FSfHebuNjam7dmzs0HwbowRn06CizcKs3jV44Y0TTzv0YiOfne38LLKNV+YWIFzj28An4n/U6npBmTIl+8sBtvaXY2w22bvBVkrRmVj6vTt3dTm5ghDCq4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2017 06:47:14.0935 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR05MB3137 Subject: [dpdk-dev] [PATCH v2 0/2] mlx5 high latency observed on send operations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Aug 2017 06:47:18 -0000 from sagi@grimberg.me: When measuring latency when running a latency critical workload on mlx5 pmd drivers we noticed high latency can occur due to delayed doorbell record update flush. This can be reproduced using the simple program [1] against testpmd macswap fwd mode. This utility sends a raw ethernet frame to the dpdk port and measures the time between send and the received mirrored frame. This patchset guarantees immediate doorbell updates visibility by making the doorbell a non-cacheble memory. In addition, we relax the memory barrier for dma-able memory. Without this fix the tsc delta was 3550760-5993019 cycles (which translates to 2-6 ms on 1.7 GHz processor). With the fix applied the tsc delta reduced to 17740-29663 (wich translates to 9-17 us). on v2: * replace compiler barrier with rte_io_wmb. Shahaf Shuler (2): net/mlx5: replace memory barrier type net/mlx5: don't map doorbell register to write combining drivers/net/mlx5/mlx5.c | 2 ++ drivers/net/mlx5/mlx5_rxtx.h | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) [1]: /* * compiling: gcc test.c -o test * run using: ./test */ #include #include #include #include #include #include #include #include #include #define BUF_SIZ 1024 static inline uint64_t rte_rdtsc(void) { union { uint64_t tsc_64; struct { uint32_t lo_32; uint32_t hi_32; }; } tsc; asm volatile("rdtsc" : "=a" (tsc.lo_32), "=d" (tsc.hi_32)); return tsc.tsc_64; } int main(int argc, char *argv[]) { int sockfd; struct ifreq if_idx; struct ifreq if_mac; int tx_len = 0; char sendbuf[BUF_SIZ]; struct ether_header *eh = (struct ether_header *) sendbuf; struct sockaddr_ll socket_address; char ifname[IFNAMSIZ]; int values[6]; struct ether_header expected; uint64_t payload = 0xB16B00B5; uint8_t buffer[1024]; int result; uint64_t before_rcv; uint64_t after_rcv; uint64_t delta; int numbytes; if (argc != 3) { fprintf(stderr, "device name and dest mac\n"); return -1; } strcpy(ifname, argv[1]); result = sscanf(argv[2], "%x:%x:%x:%x:%x:%x", &values[0], &values[1], &values[2], &values[3], &values[4], &values[5]); if (result != 6) { fprintf(stderr, "invalid mac\n"); return -1; } /* Open RAW socket to send on */ if ((sockfd = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))) == -1) { perror("socket"); } /* Get the index of the interface to send on */ memset(&if_idx, 0, sizeof(struct ifreq)); strncpy(if_idx.ifr_name, ifname, IFNAMSIZ-1); if (ioctl(sockfd, SIOCGIFINDEX, &if_idx) < 0) perror("SIOCGIFINDEX"); /* Get the MAC address of the interface to send on */ memset(&if_mac, 0, sizeof(struct ifreq)); strncpy(if_mac.ifr_name, ifname, IFNAMSIZ-1); if (ioctl(sockfd, SIOCGIFHWADDR, &if_mac) < 0) perror("SIOCGIFHWADDR"); /* Construct the Ethernet header */ memset(sendbuf, 0, BUF_SIZ); /* Ethernet header */ eh->ether_shost[0] = ((uint8_t *)&if_mac.ifr_hwaddr.sa_data)[0]; eh->ether_shost[1] = ((uint8_t *)&if_mac.ifr_hwaddr.sa_data)[1]; eh->ether_shost[2] = ((uint8_t *)&if_mac.ifr_hwaddr.sa_data)[2]; eh->ether_shost[3] = ((uint8_t *)&if_mac.ifr_hwaddr.sa_data)[3]; eh->ether_shost[4] = ((uint8_t *)&if_mac.ifr_hwaddr.sa_data)[4]; eh->ether_shost[5] = ((uint8_t *)&if_mac.ifr_hwaddr.sa_data)[5]; eh->ether_dhost[0] = values[0]; eh->ether_dhost[1] = values[1]; eh->ether_dhost[2] = values[2]; eh->ether_dhost[3] = values[3]; eh->ether_dhost[4] = values[4]; eh->ether_dhost[5] = values[5]; /* Ethertype field */ eh->ether_type = htons(ETH_P_IP); tx_len += sizeof(struct ether_header); memcpy(&sendbuf[tx_len], &payload, sizeof(payload)); tx_len += sizeof(payload); /* Index of the network device */ socket_address.sll_ifindex = if_idx.ifr_ifindex; /* Address length*/ socket_address.sll_halen = ETH_ALEN; /* Destination MAC */ socket_address.sll_addr[0] = values[0]; socket_address.sll_addr[1] = values[1]; socket_address.sll_addr[2] = values[2]; socket_address.sll_addr[3] = values[3]; socket_address.sll_addr[4] = values[4]; socket_address.sll_addr[5] = values[5]; memcpy(&expected.ether_dhost, &eh->ether_shost, ETH_ALEN); memcpy(&expected.ether_shost, &eh->ether_dhost, ETH_ALEN); expected.ether_type = eh->ether_type; /* Send packet */ if (sendto(sockfd, sendbuf, tx_len, 0, (struct sockaddr*)&socket_address, sizeof(struct sockaddr_ll)) < 0) { printf("Send failed\n"); return -2; } before_rcv = rte_rdtsc(); while (1) { numbytes = recvfrom(sockfd, buffer, BUF_SIZ, 0, NULL, NULL); if (numbytes <= 0) continue; after_rcv = rte_rdtsc(); if (memcmp(&expected, buffer, sizeof(expected)) != 0) continue; if (memcmp(&payload, &buffer[sizeof(expected)], sizeof(payload)) == 0) { break; } } delta = after_rcv - before_rcv; printf("RTT is %lu tsc \n", delta); return 0; } -- 2.12.0